“I've scaled engineering teams in various stages of euphoria and horror.”
This is Tim Howes, the closest thing to a startup green thumb there is. As co-founder of Loudcloud, he helped expand the team to 650 in just 18 months. Its meteoric rise was cut short when the bubble popped in 2001, but he was undeterred. He pivoted the company to become Opsware, where growth felt more like a “long, slow, painful climb” than a rocket launch, but ended in a successful $1.6B acquisition by HP in 2007. And after his last startup, RockMelt, was bought by Yahoo, he continued growing the mobile engineering team there from 150 to 350.
Put simply, he's seen it all. He's built products in booms and busts, hired hundreds of engineers and onboarded whole teams of acqui-hires. He's started engineering organizations from scratch and inherited massive teams (and their good and bad habits). Through all of these experiences — and more than a few missteps — he's learned how to keep things on track.
At First Round's recent CTO Summit, Howes shared the most impactful of these lessons. These include his five dimensions of scale — the five areas that engineering leaders need to nail if they want to be as effective at 50, 100 or 500 people as they were on day one.
PEOPLE: Become an Expert on Your Team (Especially Its Weaknesses)
As your team grows, it's increasingly important that you understand its psychology and how to deploy your human resources effectively. Leading an organization of 10 people is very different than leading an organization of 100 people, and Howes has identified three key shifts in your personnel needs that you should watch out for:
The oracle vs. the prophet: A lot of technical organizations have that one engineer — that one who’s been there since the beginning. “He's an oracle, right? He's got the beard, the long hair, sits in his cube. People come in and ask him advice,” Howes says.
Those wise senior team members can be invaluable, no doubt, but as new people join, that one engineer is not going to be as helpful. “Not everybody's going to come to the oracle anymore. You also need people who are more like a prophet.” Larger organizations benefit more from the evangelical style of someone who’s willing to get out there to share and impose new ideas, not sit back and wait for people to come to them.
The firefighter vs. the fire marshal: From the earliest days, startup engineering organizations get very good at putting out fires — they have to. And the people who respond best and fastest under pressure are rewarded for it. If you want to get out of catch-up or reactive mode, though, you need to start shifting your values. “I'm not saying the firefighters are setting the fires,” Howes says, with a chuckle, “But as an organization matures, you understand the need for a fire marshal.” As you scale, find and elevate people who are looking for and eliminating the causes of fires, not just putting them out.
Dennis Rodman vs. cancer: “I think it was Phil Jackson who said, ‘There can only be one Dennis Rodman on the team.’” He’s quirky, he’s weird, he’s friends with North Korean dictators. He puts his heart into the game, but he’s doing his own thing — he’s not bringing the team down, even though he doesn’t follow the rules himself. Contrast that with an engineer — any engineer, no matter how good — who has a bad attitude, complains to others, and generally brings down the team. That is a cancer that can spread quickly, and you have to cut it out.
What you want are engineers who behave more like cancer cells with great habits and skills that seem to be contagious and replicate. They actively teach others, their techniques spread, and your organization is actually better off for it, especially at scale. When you see someone with this type of influence, reward them and give them a platform to do even more.
There may well be room in your organization for all of these roles; you’ll always need engineers who will step up to get the release out the door in a pinch, and every company needs wise elders. Just be willing to reevaluate your needs — and adjust your hierarchy — before you think you need to.
HIRING: Don’t Ignore Your Biggest Investment
Howes recounts a particular moment at Opsware when he realized that nearly every engineer he’d hired in the previous three years had quit or been fired.
“The core engineering team at Opsware had also come through the crucible of Loudcloud, and it was like they’d done three tours in Vietnam,” he says. “They were fantastic engineers, and very loyal — but they were hardened and repelled new hires.” Howes learned that failing to focus on every stage of hiring, right down to training and integration, could result in a lot of wasted time with people who didn't pan out.
A strong team starts with this attitude. In Howes’ experience, the process that needs the most attention — hiring — is the one companies too often put on autopilot. “From getting resumes or referrals in the door to phone screens and interviews to integrating employees, the first thing you need to understand if you don't already, is that something along the way is always broken. I have literally never seen a hiring process work well for more than a week at a time,” he says.
So if you’ve already learned this lesson the hard way, don’t worry about it — manage it. As your company grows, what you say is going to resonate less than what you do. Make hiring a talking point in team meetings or at all hands meetings, and demonstrate, personally — no matter where you are in the management chain — the value you place on it.
As an engineering leader, you not only need to make hiring a priority for yourself, but for everyone on your team. The best way to do this is to lead by example and continually bring it up again and again.
You have to give people permission to spend time going to interviews, doing phone screens, and so on. It's extremely important.
People don't come into a job knowing how to hire. You have to very intentionally set them up for success. “You should treat it like any other skill and really train people for it,” Howes says. He very quickly learned that, left to their own devices, interviewers tend to gravitate toward the same questions. “I started sitting in on interviews just to observe, and everyone was asking about Java skills.” Now, he advises leaders to get ahead of redundancy with structured interview panels. One person focuses on Java, another on algorithms, a third on company culture, and so on. “Give everyone in the hiring loop a job during the process,” he says.
On the plus side, the more attention you pay to hiring, the more it will reveal about your organization. “There's a tremendous amount of good data in your hiring funnel. I think far too often people ignore the data and rely on hunches when things go wrong. But if you have data, you can really take action in the right places,” Howes says.
Perhaps scarier than hires quitting, though, is getting stuck with a “meh” hire. You’ve probably seen this one more than once: Your interview panel is sitting around a table, discussing a candidate. No one is eagerly advocating for him, but no one is strongly opposed to working with him, either. That’s a “meh” hire.
Larger companies, Howes notes, have tried to deal with this through official programs. “At Yahoo, there was something called the hiring committee, which doesn’t directly interview candidates but reviews their files during the process. Based on a resume and the feedback they see from interviewers, they can say, ‘Nah, I’m not feeling it. I don’t like this person.’” Amazon has a similar role they call a ‘bar raiser.’ They're basically there as a gut check for the organization. Of course, as a hiring manager trying to build a team, Howes has been frustrated by that system more than once — but he's always understood its value.
The ‘meh’ hire is death to an organization. You end up with mediocrity everywhere.
Of course, all of this assumes that you're dealing with a traditional interview process. What about growth through acqui-hires?
According to Howes, this can be a great way to build an engineering organization; you get teams of people who already know how to work together, ready to go. But there are some caveats: For starters, you can — and should — still have an interview process in these situations. “Apply the same rigor that you use in your regular interviewing process. I think too often leaders hire this way because they like the company and they like the founders. Try to create that same level of rigor in assessing every person you would be bringing on board.”
Careful integration is doubly important in the case of acquisition, for the new team and the one it’s joining.
“You really have to know what you’re going to do with people when they land,” Howes says. “And you need to understand the psychology they’re bringing to the table. Are they excited to be there, or do they feel like their own startup didn’t quite make it?”
Compensation can be sticky in these scenarios, too: Typically the newly hired group will be coming in with fat packages, which can create inequity with the group that’s been hired through normal channels. Then at the next cycle of raises, when you try to even things out, the imbalance (and frustration) gets reversed.
Howes will be the first to admit that there aren’t easy solutions for these issues. Just remember that hiring doesn’t end when new engineers are in their seats, and keep an eye out for hiccups in your integration process — if you spot dissatisfaction early, you can intervene before it becomes critical, ideally with transparency and honesty about what's going on.
ORGANIZATION: Cultivate Order Early to Bypass Chaos
From prophets to fire marshals to your own Dennis Rodman, the more people you have, the more crucial it is that you organize them consistently. For instance, when it comes to scaling organizations, Howes prefers vertical teams.
“In the 18 months I was at Yahoo, we grew from about 150 people to 350. The only reason that we could do that and continue to ship products and maintain any sense of sanity, was because we had many small teams that shipped mostly independent products,” he says. Equip each team with everything it needs to get a product off the ground, from engineering to QA to product management to marketing. By doing so, you can beat out a lot of the unwieldy, big companies.
This isn't to say that vertical is the only way to go. Horizontal organization has proven effective in other situations. Whatever your structure, though, the important thing is to formalize the role of management. Create the role of tech lead and fill it thoughtfully.
Over 50 people, if you're not formalizing roles, everybody thinks they're a tech lead. And that gets bad.
Watch out also for the temptation to take your top coders and make them managers. In Howes’ experience, it's usually not the right fit. “They're different skills. The thing to realize is that management is about people, it's not about code,” he says. If someone comes to you with a desire to transition into management, by all means work with them on it. But don’t push.
“People often make an uncomfortable transition to management because they think that it’s the way to get ahead. But it's quite often the wrong thing to do.”
Lastly, as your company grows don't stop releasing fast. “There’s a big tendency for things to slow down and grind to a halt,” Howes says. “But whether you’re in mobile, web, whatever you’re doing, the most valuable thing you can do is prioritize shipping fast.”
The secret to success in the world of rapid releases is to never lose sight of the technical debt you accrue in the process. “You need to be able to pay that back explicitly, and you need to think about those trade-offs,” Howes says. “You can get to market fast or you can optimize for performance.” If you do build a “release fast” culture, be sure that you’re also formalizing a mandate to go back and clean things up from the beginning.
COMMUNICATION: Make Peace with Repeating Yourself (Ad Nauseam)
In many ways, communication changes are the easiest ones to spot. You can’t miss the move from a founder’s living room to a real office, or from one floor to two. At a certain point, you’ll reach a size where coworkers won’t all know each other. “The thing to realize about communication is that it's what builds trust,” Howes says. Lose sight of it as you grow, and you may find yourself facing a critical trust deficit.
At 10 people, team communication is interactive. You can spend less time crafting your message because you’re delivering it in a dynamic fashion. “Communication happens in conversation, it’s higher-bandwidth. One conversation goes a long way,” he says. Grow to 50 or 100 people, though, and this changes drastically.
Communication is no longer interactive; it’s one-to-many. “The consequence of that is you need to spend much more time crafting your message. You need to be crystal clear, because you get nothing back from people.” Clarity often needs a helping hand from repetition. “You'll think to yourself, ‘Oh my God, how did you not understand this from what I said at the all-hands last week?’”
You'll be surprised how many times you have to repeat yourself to make your communication stick.
But scaling communication isn’t just about your dispatches to the team. As growth picks up speed, you need to foster regular, healthy interaction among your employees, too. Howes recommends thinking of this in terms of both formal and informal opportunities to connect. “Formalize communication with things like architecture reviews, code reviews, and demo days. All of that stuff helps the organization know what everybody's doing.”
But pay attention to informal communication, too. Once your organization passes a dozen, the whole team isn’t having lunch together anymore. So schedule brown bag sessions or beer bashes, and think outside the box for creative ways to inspire meaningful interaction. “At Yahoo, one of our managers created something called Three at Four,” Howes says. “It was three talks by three people, two technical and one nontechnical, at four o'clock every Friday. And it was great. It really worked to get people together and thinking about communicating with one another.”
QUALITY: Don’t Leave It to Chance
If you’re scaling your engineering team, it’s because either your product is getting bigger or you’re adding new products. Either way, you’ll be juggling many different teams working on many different things, and keeping a close eye on quality will become exponentially more challenging.
“When I came to Yahoo, one of the first things I saw was that we had mastered the art of shipping fast,” Howes said. “And our products looked really great. But they were still much crashier than we wanted.” So he hit the ground running, establishing a culture of quality with three key rules that he recommends to every engineering leader:
- Pay attention: Sometimes, the most effective advice is just that simple. “Make a dashboard. Start every product review with ‘How's your quality?’ Send an email to anyone who had quality problems that day or that week,” Howes says. When he started paying attention to and fastidiously recording details like crash rates — and made it clear that he was doing so — the rest of the organization quickly responded with safeguards.
- Make a release checklist: Of course, there are a lot of benefits to managing an ever-increasing team of engineers — for one, they bring an ever-increasing pool of knowledge. Howes recommends codifying that knowledge into a checklist, a set of boxes each product team must check before any product can be released. “What's your binary size? What's your memory usage? How long can you sustain a monkey test without crashing? How long is the startup time, cold and warm? What have you done on security? What external URLs are you using? There were literally 200 items on our list,” Howes says. “Then we started automating tools to help each team get through this list so it wasn't such an arduous process.” Standardizing this list will have a major impact on the quality of your releases, and make each team’s lessons available to the rest of the organization.
- Implement reviews and automated testing: Code reviews, functional tests, unit tests — they all ultimately serve the same purpose. “They force you to structure your code well. I think that's the primary purpose of unit tests and functional tests,” Howes says. “And nothing is better for keeping engineers honest and paying attention to code quality than knowing their code will be reviewed by their colleagues.”
Still, when it comes to the details, he has a word of caution: “Don’t get bogged down in religion.” What does this mean? Basically, it’s tempting to try to find one true coding style that works for your team, or to keep researching tools until there’s a unanimous favorite. “In my experience more than half of every group hates any tool that you're likely to use or any coding convention you adopt,” Howes says. “It’s more important to just pick one and move on.”
In 20 years, at 6 companies, and through working with hundreds of engineers, these are the five areas Howes has seen make the biggest difference — often the difference between success and failure. But this advice isn't meant to represent a formula or a panacea. The best engineering leaders will always make missteps and will always take the extra steps to learn from them, he says.
“There isn't really a one-size-fits-all solution. Things will always break. The real key is to stay self-aware, all the time, so you can identify those broken things and address them. Scaling is a constant process of reinvention.”