When it went public in 2011, over a decade after the company’s founding, Pandora employed fewer than 40 engineers. With this skeleton crew, the company built products for 70 million monthly users on the web, iOS, Android, Windows Phone, a thousand consumer electronic devices, and in over 100 types of cars. It also generated half a billion in revenue — laying the groundwork for its $7 billion valuation today.
Compared to Twitter and Facebook, with their armies of engineers, Pandora is one of those rare Hail Mary success stories that keeps entrepreneurs and investors betting on long-shots. From day one, it pushed against constraints that these other companies didn’t have. Namely, the fact that it had to hand a huge chunk of its funding over to the music industry, leaving it with a scrappy budget that forces companies to stay lean and get creative.
That’s how the Pandora prioritization process was born. In an exclusive First Round CTO Summit talk, Tom Conrad — the company’s CTO since 2004 — broke down how the company figured out the exact right things to build fast and literally demanded buy-in from stakeholders (albeit with fake money, as you’ll see).
Turning Constraints into Opportunities
When Conrad arrived at Pandora (then Savage Beast Technologies, a maker of kiosks for brick-and-mortar music retailers), his first steps were to understand the company’s limits and the resources he had to expand them. Coming from Apple and a handful of tech startups, he understood the nature of time and form-factor constraints. As Pandora’s early success grew, the money was the sticking point: With all the revenue going out to rights-holders, there was less to invest in the technology and product than anyone at the company would have liked.
This meant Conrad and his team would have to move fast. But also incredibly carefully.
Every single day, we had to do the work that would matter most, and we couldn't afford to make big mistakes.
He readily busts the myth of “failing fast.” This wasn’t the environment at Pandora, with little time and money to waste. “There’s this motto in our industry that says fail fast and fail often,” he says. “But none of us can actually do that right? We have all kinds of constituents — employees, investors, users — they are expecting you to do smart things, not dumb things. So one of the first things I said was let’s not pretend we can just try things and some will work out and some won’t. That’s not winning, that’s losing.”
That’s also a ton of pressure, especially on a small staff. The good news is that you learn new things about your business every day. That’s why Conrad also threw out the concept of long product roadmaps. In those early years, the environment was far too dynamic to predict what would happen in six month, much less a year. This, he says, is true of every young and even not-so-young startup.
“A week from now, you’ll be smarter about everything — about what your users want, how to monetize it, the expectations of your investors,” Conrad says. “You absolutely can’t commit to long-term roadmaps. A lot of people will ask you for one, and they have good reasons — people who have given you money, advisors, employees, potential employees. But you have to tell them you’re not going to answer their questions. The more you talk about what you’re going to do a year from now, the more that becomes etched in stone. You have to make sure you’re doing the most valuable work at any given time.”
It’s almost certain that what you think is important today won’t be important at all, even a quarter down the line. Adopting this attitude — born out of constraints — will keep you nimble, keep you focused on the ripest opportunities, and make the best use of everyone’s time.
Every 90 Days
90 days is the length of one quarter. That’s how far you can reasonably think and plan ahead when you’re in hyper-growth, Conrad says. And there's a question you have to ask yourself at the start of every quarter:
What would be stupid for us not to do in the next 90 days?
“This is not about all the cool things we think we might want to do. The bar is, someone has to be able to articulate why the company would be dumb not to invest in a particular idea or feature in the next three months,” he says.
You can make this call for ideas as broad and wide as you want. Especially if you’re still at five to ten employees. You can include everyone in this phase of the process — individual engineers, the people that work with your business partners, the people in community or customer service who hear what your consumers are saying every day.
“The important thing is that you’re not trying to impress anyone. This shouldn’t be a competition for how many ideas people can think up. The person evangelizing for an idea has to truly believe it is vital for the company right now,” Conrad says.
At Pandora, this step would usually generate between 60 and 80 ideas, but there would be no way the company could pursue all of them. This is where serious, systematized prioritization became core to the culture. How do you go from 80 ideas to a reasonable number of features that less than 40 engineers could build in 90 days?
It started with making one presentation slide for each idea. Each slide would have a heading and several bullet points defining the scope of the feature and metrics describing what success would look like and how it would impact the business. That’s all people got to make their case. And given the speed of the whole process, there was little time for precision.
Especially as a CTO, Conrad says, the important thing is being able to roughly estimate how much work something will take and how much engineering capacity is required to ship something.
To make this easy for everyone to grok, Pandora decided to express scope in terms of dollar amounts.
“Basically, a feature that takes one engineer one month to complete is worth $5. If it takes them two months, it’s $10, $15 for three months. So if you’re thinking about dedicating an engineer to one project for the whole 90 days, that’s $15. If you’re going to put two engineers on it, that’s $30 and so forth,” he explains. “With this in mind, we put dollar values on each of the idea slides we made.”
For Conrad, it was important that prioritization wasn’t slowed down or stalled out by team leads going back and forth to their engineers, analyzing capacity and scope. Bottom line, he wanted to end up with a list of the features the company had to build to remain competitive. Without rudimentary analysis and quick decision-making, this wouldn’t be possible.
“You literally tell yourself, okay, we have 10 developers on the team. For three months, at $5 per developer per month, that gives us $150 of engineering capacity,” he says. “That way you can also factor in things like an engineer whose going to be out for a month. Also, when I get asked what $5 really means, I say it’s an engineer who works on a project, focusing solely on that, but it doesn’t ignore that they do go to meetings and they do get distracted at times. So you have to be really honest with yourself about how how much an engineer can practically get done in a month.”
The last step of this phase of the process is to pick your prioritization team. Even at the earliest stage, you probably don’t want to include everyone. Most early-stage startups aren’t democracies; and the companies they grow into certainly aren't. It’s best not to pretend they are.
At Pandora the team consisted of the executive staff. At the start, that meant the CEO, founder Tim Westergren, the VP of Business Development, and the Chief Revenue Officer would pick which features got built. Later on, the team expanded to include the CFO, the Chief Council and the VP of Human Resources.
“It’s up to you who you let into the room,” Conrad says. “But you want them to be people who have the ability to pop up to the highest level and think about the health of the whole business. They have to really feel that obligation to the whole company — not to an individual team or functional area — the whole company.”
Get Sticky
This is where Pandora would go analog.
Every quarter, Conrad would print out and tape up all 80 submitted idea slides with their requisite dollar amounts. Then he’d hand out stacks of Post-It notes to each of the stakeholders in the room.
Taking his example, if you have five people participating in the prioritization process and $150 in engineering capacity, you would make five stacks of stickies, each worth $5. Everyone gets six so they have $30 to “spend” on features. It’s important everyone gets the same amount whether they’re the CEO or customer service.
“This is a pretty amazing moment because it’s usually the first time these people get a sense of just how much demand there is for new ideas and features within the company,” Conrad says. “They each went in thinking their ideas were clearly the ones that would be stupid not to accomplish, but when they see everything they think, ‘Wow, there’s a lot more out there that we could be doing.’”
Then comes the hard (and fun) part. All of these people make the rounds, reading through the slides, and “buying” the features they think are most valuable by applying their $5 stickies.
Every time during this first pass, at least half of the slides ended up without a single Post-It note.
“This is incredible, because someone very smart at one point thought, ‘We would be absolutely stupid not to do this thing.’ But really, when viewed in context of all the opportunities for the business, half of the things people thought were important immediately fall away," says Conrad.
On the other hand, there were a handful of features that immediately get fully funded. If there was a $20 item that had four $5 stickies on it, that became a defining focus for the company over the next quarter. Then there might be a $25 item that earned $20 but people still agreed it probably wouldn't or couldn't happen.
“You get into this collaborative consolidation, re-scoping with a little bit of horse-trading,” Conrad says.
Force yourselves to get everyone on the same page about what's important for the business right now.
More often than not, people decide to re-distribute their stickies based on this dialogue. A $30 item with only $5 stuck to it might end up getting fully funded if someone advocates for it strongly enough.
“By the end, this really interesting, magical thing happens,” Conrad says. “We would get to a point where only about 20 features approximating our capacity were left standing. Most things just naturally got left on the cutting room floor.”
At Pandora, Conrad and his team used this process four times a year for eight years. “Every time this miracle would happen where after just two or three hours of work, we’d emerge with 20 out of 80 things to do. It would be just the right amount, and everyone would be bought-in enough to evangelize it to their teams. I think it’s the idea of buying power. Because people had to spend their own ‘money,’ they were never left feeling like they were forced to build things they didn’t believe in.”
Time to Deploy
Pandora would start its prioritization process two to three weeks before the start of each quarter to give the development team time to finish the features from the previous cycle. After the new 20 items were picked, there’d be a mad dash to flesh out their specs so that everyone would have an informed perspective on the next 90 days of work.
“Two things were critical: that the leadership really advocated for and supported the notion of this whole process, and that we institutionalized the understanding that we had limited resources,” Conrad says. “We created this culture around how we prioritize, especially the idea of how quickly we would do it and get things done. We didn’t want people to feel like their ideas would never happen.”
Speed never stopped being important. As quickly as the company could take on new tasks, its market and opportunity were changing just as fast. Nothing made this more apparent than starting a new prioritization phase. It was critical for Conrad to never start with the old ideas that got cut previously. Everyone needed to start from scratch.
“When you build the list all over again, and then compare it to the 60 things you left on the cutting room floor last time, you really see how much things change,” he says.
It turns out 50 of the things we didn't do, no one thinks are important anymore. That's when you step back and say wow, we're not ready for a long-term roadmap.
Instead, the team would come up with a fresh set of 60 to 80 things that many of them had never talked about in-depth before. “To me, this was an indication that we needed to keep this process going for as many years as possible.”
Eventually, the company did grow out of its 90-day cycle. And it started coming out with longer and longer roadmaps. In short, it grew up. But a lot of things had to come together first.
“Two years ago, after the IPO, we started building out a true engineering organization divided into teams that needed to manage their own backlog and their own prioritization — that forced a revision to our process,” Conrad says. “Also, as a publicly-traded company you have to do a lot of long-term planning with respect to revenue. You have to become more predictable for investors. And revenue is tied to products that have to ship to meet projections.”
Scaling didn’t only mean more engineers, either.
“The bigger we got, the more often exec staff didn’t have the necessary perspective and insights to make snap prioritization decisions,” he says. “We had to force expertise down through the organization so that teams could make their own calls. This is much harder. And still, today, we only have about 80 developers.”
In short order, Conrad has ended up running a very different organization than he stepped into. Yet, despite all the perspective he’s gained from Pandora’s growth and globalization, he believes the company’s success really stemmed from its early ability to pull the trigger on what needed to be done in the moment.
“I think our results are very closely tied to making really, really good priority decisions,” he says. “I have a hard time looking back on what we built and pointing to the things that I would say were a waste of time or a bad investment.”
The Breakdown
For anyone who wants to implement the Pandora Prioritization System, here are the steps:
1) Before every quarter starts, pose the question: “What would be stupid not to do in the next 90 days?”
2) Define and scope every idea on a single presentation slide.
3) Roughly calculate the engineering capacity needed for each of the ideas generated, and how much you have to leverage total.
4) Express this scope in dollar amounts: $5= the amount of work one engineer can do in one month. $10= two months. $15= three months. $30= two engineers for all three months. If you have 10 developers working for three months, you have $150 to play with. Include a dollar amount on each slide.
5) Pick a small team to participate in the prioritization process. Choose people who have the whole company’s best interest in mind.
6) If you pick 5 people, make 5 stacks of Post-Its with $5 written on them. Hand the same number of Post-Its to each person based on your engineering capacity.
7) Print out all your idea slides and hang them on the wall. Talk your prioritization team through each one.
8) Have your team “buy” the ideas they believe are the most valuable to the business by sticking their Post-Its on the corresponding slides. People can choose to spend more than $5 on one idea.
9) Discuss the allocation of stickies, consolidate and re-scope.
10) A miracle occurs — you end up with a manageable number of tasks that can be executed with the resources you have, and you have buy-in from leadership in only a few hours.
11) Develop specs, scrum, and make sure everyone in the company knows how these decisions were made. Make prioritization a part of your culture.
12) Repeat 90 days later. From scratch.