Veteran CTO (with Multiple Successful Exits) Answers Your Top Startup-Building Questions

Last week, we ran a piece with Adil Ajmal, CTO of LendingHome — the 300-employee strong startup remaking the mortgage industry — in which he focused exclusively on how to hire (and close) the best engineering candidates for your team.

It struck us exactly how much tactical advice Ajmal has to offer startups when it comes to building and managing strong engineering teams. The man’s experience is amazing. We don’t know anyone else who has grown teams from tiny to gigantic across 7 companies. He did it at TenMarks as CTO and VP Engineering until it was acquired by Amazon; at Posterous before it was snapped up by Twitter; and at Homestead before it joined Intuit — and more still. Now at LendingHome, product, design and data science all fall under his purview, in addition to engineering.

So, to extract even more of his wisdom for the benefit of the entrepreneurs in the First Round community, we had him do an Ask Me Anything (AMA) session on our intranet, First Round Network. Bowled over by the quality and specificity of his advice, we wanted to share it with the broader startup ecosystem. Below, we’ve republished Ajmal’s AMA (questions submitted by a number of engineering leaders in our portfolio) and all of the advice he offered in response to the 12 toughest questions that we haven’t just heard here, but from many founders and technical leaders for years.

How do you measure the performance of software engineers at an individual level, and how do you measure performance at a team-wide level?
What are the roles and responsibilities on your engineering team? Do you have designated software architects? Tech leads?
What’s your process for product prioritization?
How can design fit ideally into the product development process? And how do you get design involved early in a project without wasting their time?
What commonly-held beliefs about building engineering teams are just plain wrong?
What are the steps of your engineering interview process and why are they important?
When does it make sense to hire a VP of Product, and what do you look for?
What frameworks do you use for managing technical debt?
How do you decide whether something is so painful that it merits a full re-write?
How does your technical organization work with the other operational components of the company?
Have you used coding tests as a part of your interview process? If so, which and why?
Can you share advice for an engineering manager hiring people with more experience than themselves?

1. How do you measure the performance of software engineers at an individual level, and how do you measure performance at a team-wide level?

While the work that an engineer does may be different than someone in another function such as design, marketing, product, etc., the fundamentals of performance management remain the same. This also applies to measuring performance of teams regardless of function. While there are many frameworks for managing performance, to me it boils down to two fundamental things:

Did the person do what they agreed to do or were expected to do? (The “What”)
How did they go about doing it? (The “How”)

The most important pre-cursor to any performance evaluation is mutual agreement on what is reasonably expected of that person. This includes both explicit and implicit pieces. The explicit part may include a project deliverable by a particular timeline based on certain requirements, while the implicit part may include things that are expected of them in their role regardless of the project.

For example, for an engineer this could be writing high-quality, well-tested code. And so while you may go into a lot of details about the project, you’re not required to remind them that their code should be of a high standard. That part is just implicitly expected. You should clearly state and get agreement on both explicit and implicit expectations when someone starts a job, when their job changes, and during performance check-ins.

The part about expectations being reasonable is key. For instance, you can’t reasonably expect the same level of work from people with significantly varying experience levels. You may expect a seasoned engineer to be able to lead a large project, independently design and build a complex and scalable system, etc. At the same time, you shouldn’t reasonably expect a junior engineer to be leading the same type of large or complex projects.

So, if a seasoned engineer doesn’t do well on a large and complex project, then they may be underperforming. On the flipside, if a junior engineer really steps up and does a great job of what would be expected of a significantly more senior engineer, then they are exceeding expectations. I see many people forget to keep this in mind, and miss out on helping their employees grow. When you’ve both agreed on clear expectations, it becomes easier to measure their performance on both the “What” and the “How.”

If they were developing a feature, then the “What” should include things like:

Did that feature work well in production?
Was it built to spec?
Did it launch on time?
Did it deal well with scale?

The “How” should include:

Whether the person collaborated well with the rest of the team?
Did they take short cuts and build a system that won’t be maintainable?
Did they follow all the right steps and build something extensible and long lasting?
Did they break someone else’s system along the way?
Did they teach others while doing their own work?
Did they learn something new along the way?
Were they innovative and solved the problem in a new and novel way?
Did they just force fit a previous solution to the new but different problem?

A lot of managers assessing performance focus just on the “What” — but the “How” part is equally if not more important.

I would never rate an engineer (or any individual for that matter) very highly if their deliverables were great but they burnt through a lot of people or systems to get there.

You can measure a team’s performance in similar ways. In particular, ask:

Did they deliver what they were expected to deliver?
Did they do it in such a way that you would want to work with them again on similar things?

2. What are the roles and responsibilities on your engineering team? Do you have designated software architects? Tech leads?

I’ll start by saying that I’m not a fan of having a designated architect role or an explicit architect title on a team. I believe that architecting a system is the responsibility of all engineers who have to build and maintain that system and not the exclusive responsibility of a single individual or a team of individuals who just tell others what to do without actually building the system themselves.

I know this may annoy my architect friends, but oh well. I’ve worked at large companies on both sides of this issue. For example, Intuit has a very strong architect community with a mature architect career path, and I know some great folks in that cohort — including the really awesome guy who is now the company’s chief architect. On the flipside, there's absolutely no architect job at Amazon. As an engineer, you could be on the IC career path or the engineering management career path, but either way, you're responsible for architecting really large and complex systems.

Every organization that I’ve built or run follows the latter path. Please don’t read this as architecture by committee or consensus — both of which I’m against. Someone still has to take the responsibility of leading architecture. But I expect that person to be a code contributing member of the engineering team.

Now about tech leads:

‘Tech lead’ should be a function or responsibility that one plays and not an explicit job level.

To me, it means leading other engineers on a project (not people management) both in terms of the technical decisions as well as some coordination of the work between individuals.

I don’t believe that you have to be at a certain level to be a tech lead on a project. For example, even a first-year engineer could be leading a project with one or more engineers or interns if they have the ability to do it. I’ve also had some really awesome seasoned engineers on my teams who are thought leaders and subject matter experts but don’t want to lead people. So, while senior people end up being more of the tech leads, it’s not a job level in itself.

At LendingHome, we’ve defined different levels across our organization and defined roles and responsibilities for each level. We’ve tried hard to remain consistent with the larger industry here so that our levels and titles can be easily translatable to other good tech companies. We have two paths in our engineering org: one for ICs and one for engineering management. You don’t have to become a manager to continue to rise in the organization.

Our engineering levels are Engineer, Sr. Engineer, Staff, Principal, Sr. Principal, Distinguished and Sr. Distinguished. Starting at the Staff level, they run parallel to Manager, Sr. Manager, Director, VP and SVP. You could be at any of these levels and play a lead architect or tech lead role. In fact, I expect a tech lead to lead architecture for their project. In terms of org structure — which is different than levels and roles — we have the engineering and product team broken down into three high-level focus areas:

Acquisition and Customer Experience
Origination Platform
Investor Management

All three of these are large and have sub-teams of their own. We’ve previously tried a few different variations as well, such as organizing by business line, organizing purely around large initiatives, etc. based on the needs of the company at various life stages. I think our current org structure is effective, and I don’t see it becoming a hindrance anytime soon. That said, I’ve never known a perfect org structure. No matter how you define it, you have pros and cons.

When designing any org structure, you have to be clear on what you’re solving for and how you’re going to deal with/anticipate the inevitable repercussions.

3. What’s your process for product prioritization?

Every year, we set only a few key company-wide goals. Nested within these are goals that each of our business lines sets for themselves for a 6-month period. We then create quarterly roadmaps that are aligned to those goals. These roadmaps feed into our regular sprints. This way, everything cascades down from what the company as a whole has to accomplish over the course of the year.

Creating our roadmaps always starts with each product manager compiling a prioritized list of goals that they want to accomplish in conjunction with their cross-functional partners. We bring all this together for each of our business lines and create a stack-ranked list of things that we want to accomplish in the next quarter. This always include things that we’re carrying over from the previous quarter as well.

Once we have initial alignment on these priorities, product managers work with engineering to get T-shirt level estimates for the top projects (i.e. high-level roughly small, medium, large or X-large estimates). Engineering also provides very clear capacity for the quarter. Using these two tools, the team can hash out effective tradeoffs and create a final list of the high-level product priorities for that quarter. This is not a project plan, but rather a list of what we want to accomplish.

Our sprints then pick off from this list and continue to execute throughout the quarter. We try to do enough due diligence here so that we don’t change this list too much mid-quarter. But if things come up that require alterations, then we force ourselves to make trade-offs on the list instead of just adding more items. This is easier said than done but is extremely important. It’s vital to remember that trades are never even. For example, you can’t do an even trade with a project that’s already started because you lose a lot on that. We try to make as even of trades as possible, but it requires being ruthlessly honest and realistic.

4. How can design fit ideally into the product development process? And how do you get design involved early in a project without wasting their time?

In an ideal state, there should be design leads or point people on the design team designated for different parts of your systems and your business. These leads should be involved in projects from an early stage so that they can pull in other resources from the design team as needed throughout the project.

Our challenge in the past was that we didn’t have enough people on the design team to take up these lead/point person roles. The goal is to set up this structure so the design team can grow into it. Design should be involved as early as possible — but that moment does depend on the nature of the project. For example, we wouldn’t want to involve design in some of our engineering projects that are focused on pure backend services and infrastructure at any stage. On the flipside, design should be involved from the very start on projects that are customer-facing — especially related to brand, marketing, application flow, etc. And there are usually multiple people from the design team who get involved in these type of projects very early on.

You shouldn’t worry about wasting design's time if you minimize the number of people involved.

Worst case scenario, they’ll walk away with important context on a project. It’s critical to note that I don’t see design as purely look and feel (visual design), which can be brought in at a later stage after most other things are figured out. Design to me is a lot more holistic and involves several aspects: interaction, research, visual, etc. The best way to not waste design’s time is by involving the right folks at the right stages and not pulling everyone into every stage. For example I wouldn’t want to pull in visual design when the requirements aren’t even defined yet. But I would want to pull in interaction design as early as possible because they can have a lot more influence over defining the requirements and the solution very early on.

5. What commonly-held beliefs about building engineering teams are just plain wrong?

I’m sure there are many, but here are three that immediately come to mind:

1) Hire the smartest people and put them together and you’ll get a high-performing team. You should always strive to hire people that are smarter than you. But just bringing a group of smart individuals together doesn’t make a high-performing team. What will make it high-performing is a group of smart people who really want to work with all the others on the team and care about the success of the team more than themselves.

Solving for the latter is much harder than just finding smart people, and that’s the piece that really makes the team succeed. So you could say that being smart is just table stakes in this scenario. For the second part, you can use the STAR (situation, tasks, action, results) methodology in interviews — which is built around the premise that past behavior is the best predictor of future behavior. If someone hasn’t worked well on teams before or doesn't have a track record of being selfless, there’s high likelihood that you’ll encounter problems with them again.

2) Hire everyone that’s like you or others on your team.

I’ve wished many times that I could clone some of the awesome people on my team. Luckily, for me that option hasn’t existed.

If I could just make clones of the best person on my team, I’d end up with no diversity of thought and eventually no innovation. Diversity of experience and thought is what fuels new and creative solutions. A diverse team will always be a better and stronger team in the long run.

3) Put a smart group of people together and they’ll solve any problem. I’m sure I’ve been guilty of saying this many times as well. And to a certain extent, it’s actually true. Smart people like to solve challenging problems, and they’ll often come up with a decent solution. But that’s not what builds a strong team. The key here is to assemble people who are not only smart, but are genuinely passionate about solving the problem that you’re trying to solve. Without that sincere interest and passion, you’ll likely still get some great work done in the early stages — but in the long run, you end up with disengaged employees and a low-performing team.

6. What are the steps of your engineering interview process and why are they important?

Our process at LendingHome is pretty straightforward. As a first step, a recruiter may or may not do a phone call with the candidate. It depends on whether we’re trying to engage a passive candidate or if the person applied proactively.

The interview process itself starts with an initial phone screen that is designed to serve two purposes:

To give the candidate information about the company and role
To assess the candidate’s problem-solving and coding abilities. (We have them work through an exercise and develop a solution on CoderPad or Collabedit during this screen.)

The candidates that pass this phone screen are invited to a final on-site interview. This is a full day consisting of sessions with 6 interviewers — 3 engineers, 2 engineering managers (we pick 2 from our group of Directors, Managers, VPs, and me, the CTO) and 1 product manager (or another cross-functional partner from design or marketing). I always want a non-engineer on every engineering hiring panel to get a different perspective, and to give the candidate the opportunity to meet people from other functions who they’d work with if they join.

On site, we try to cover both functional competencies as well as leadership and behavioral skills. The technical portion covers architecture, system design, problem-solving and coding. I’m a big fan of using the STAR methodology to assess non-technical skills. The idea is that past behavior is the best predictor of future behavior.

Each interviewer on the panel types their detailed feedback into our applicant tracking system, Greenhouse. Everyone submits blind comments, which means that they can’t see other people’s feedback before they enter their own. We then conduct a 30-minute debrief for every candidate within a day of their on-site interview and make the decision to extend an offer or not.

We move on offers very quickly if we decide to move forward with a candidate in the debrief. Offers are sometimes contingent upon reference checks, but that’s usually for more senior roles.

7. When does it make sense to hire a VP of Product, and what do you look for?

This question is extremely dependent on the life stage of the company, as well as what you’re trying to solve for by bringing on a VP of Product. Contingent on the size of your company and technical organization, the VP role can imply and include different things:

In a larger org, they are probably just running product for a particular function or business line.
At a smaller company or org, the VP of Product probably should lead product vision and operations for the entire company — basically on the road to becoming Chief Product Officer.

Answering this question does rely on these specifics, but I’ll provide an answer for companies looking to bring on a first VP of Product to be responsible for managing all product functions for the company.

Early on, the CEO usually acts as a company’s first product manager. Then, as the company grows, others start to share the PM responsibility on a part-time basis. The engineering team is usually first to grow, working closely with the CEO. If there isn’t a technical co-founder, then usually the first executive hire to come down the pike is a VP of Engineering. From there, a CEO may bring on a full-time PM to do the tactical work, such as creating specs, coordinating with engineers, etc. — still based on the CEO’s product vision. This isn’t an executive role. Several of these PMs might spring up, loosely configured into a team still reporting to the CEO.

It’s at this point that a CEO should become cognizant — if they aren’t already — that they have many other responsibilities and their efforts are probably needed elsewhere. This is the moment they have to recognize they need a VP of Product.

More often than not, many CEOs have to be pushed to make this decision because they generally believe they are running product and will forever.

If you’re a CEO in this position, you have to answer one key question that requires a lot of self-awareness: Should the incoming VP of Product mainly just run the product function and manage the product managers while executing the CEO’s instructions? Or should they own the product vision and direction as well?

This is a really tough question, and the answer will dictate what type of person should be brought into this role. This should not be a unilateral decision. Crowd source opinions from your board, investors, trusted members of your team. Lean on them to craft a VP of Product role that makes the most sense for your higher-order, company-wide objectives.

8. What frameworks do you use for managing technical debt?

This is something we still need to get a lot better at. For the longest time, we grew really fast, became resource constrained, and as a result, accrued a good amount of technical debt. We’ve since re-factored several parts of our system, driven by the following attributes:

The number of issues coming out of a system as it scales.
The complexity of maintaining that system.
The need to break that system down into independent and more manageable services or systems.
The inability to do fast development on a part of the system because of too many interdependencies.
The inability for an existing system to meet the growing business complexity.

Some of the next questions that come up are:

How do you balance the need for new feature development to move the business forward with re-factoring existing systems that may not immediately show a clear business benefit?
What percentage of resources do you invest on either side?
Do you incrementally improve a system while it’s in production or do you do a full re-write and replace the old system entirely?

Over time, we’ve become better at evaluating these questions, but there isn’t always a single approach that works. For example, we currently have a major re-write project going where I wanted the team to build a new system in parallel and replace the old one — because incremental improvements on that system would have taken too long and would’ve been too painful. At the same time, we have a few other tech improvement projects ongoing that are enhancing some existing systems.

9. How do you decide whether something is so painful that it merits a full re-write?

Again, there’s no one-size-fits-all approach here, but perhaps start by assessing the following two factors to help make the call:

1) The cost of doing something incrementally vs. re-building. I generally look at cost as time and pain, but you can add other factors as well.
2) Can the business withstand the burden of you stopping new development for a period of time?

In the case of large re-writes, it doesn't make sense to add to the old system in parallel while you're building a new one — you have to stop or really slow down new development in that area. Otherwise, it not only takes almost double the resources, it keeps delaying your migration to the new system. Can your business withstand this cost right now?

Don't leave No. 2 entirely to someone non-technical to decide because it'll be very hard for them to make an educated call on this.

You, as an engineering leader or team member, have to make sure that your partners across the business understand the costs and tradeoffs really well.

For example, if you just ask them whether they'd choose between new feature development or a re-factor, the default answer will generally be new feature development, and I wouldn't fault them for that at all. It’s the answer that will probably seem most obvious from their perspective.

However, if you can really make them understand the stakes and costs of not doing the re-factor — i.e. the system will break in the next X months at scale; security will be significantly compromised and could be a company-killer; your engineers will not remain for much longer because they hate the system, you can't onboard new engineers faster; feature development pace will continue to slow; product support will continue to rise, etc. — then you can have a much more informed discussion and decision.

10. How does your technical organization work with the other operational components of the company?

We’ve built a very strong partnership between technology and the various functions across the company, including our operational components, such as Mortgage Operations, Sales, Capital Markets, etc. We’ve done this first and foremost by having a product manager who is responsible for being a strong partner to each of these functions and owning the product roadmap for that function.

For example, we have individual PMs focused on things like acquisition, mortgage operations (many sub-functions within this one with multiple PMs), servicing, investor management, and more. These PMs work closely with these teams, shadow their work, internalize their needs and pain points, and use all of that to define the right product features and priorities.

We also have weekly planning and status meetings with different functions across the company attended by PMs, engineering managers and directors, partners from those functions, and in some cases GMs for the different business lines. Part of this meeting is just a status update on what’s being developed, what just launched, etc. But the majority of the meeting is to jointly figure out the needs of that function, define solutions, and prioritize projects.

In several cases where we do large releases of internal tools, the product team also conducts training sessions for our operations teams and offers office hours. We also have a formal process for taking in-production support requests from our operations floor. If the issue ends up being technical, then the engineer working on the issue is also working directly with members of the operations team in question.

The same happens when we’re developing new features and the PMs and engineers are collaborating with members of the relevant operations team to define the right solutions. For teams that are smaller, we have Slack channels with engineers, PMs and members of those teams. This works for a while but doesn’t scale when you start getting into large teams (such as our mortgage operations and sales teams at LendingHome). Eventually, we have to roll out a more formal communication process.

All this said, the partnership between the technical and operation teams hasn’t always been perfect and has sometimes varied based on the different non-tech groups. This has largely depended on whether the non-tech group thought of the technical team as a service organization for them where they could just give them instructions, or whether they considered them a partner who could jointly solve problems with them. We always shoot for the latter, and at this time, I’d say we’re in that state or close to it. In cases where people have tried for the former, it usually didn’t work that well and the costs are steep.

11. Have you used coding tests as a part of your interview process? If so, which and why?

We do use coding exercises as part of our interview process. We are language agnostic, and the candidate can write code in any language of their choice. What we’re trying to assess is whether they:

Write good working code
Cover edge cases
Solve the problem at hand correctly
Create good tests
Take feedback if there are faults in their code

The coding exercises themselves are not that big or intense. And they’re always preceded with a problem solving portion where we assess whether the candidate can efficiently solve problems or not. Once they pass the problem solving part, we ask them to write code for the problem that they just solved.

Ideally, everyone that you hire should be better than you in at least some category.

And to me, that also goes for hiring people with more experience than you. The main things to figure out are how you will add value for them and how you will benefit from their experience.

A good starting point is being very transparent and setting the right expectations. Understand where this person will add value and where they’ll benefit from your help. For example, they may be a much stronger engineer than you and would not need any help from you in designing and building complex systems. But they may not have as much context as you about your company’s business, its customers, legacy systems, etc. You could add a lot of value for them by getting them more familiarized with all these aspects so that they can do their job better. Similarly, you may also be able to help them navigate the org and thus improve their collaboration with other teams faster.

These are points to mention to them during the recruiting process — not only after they’ve taken the job. People are convinced and comforted when these needs and interests are acknowledged.

The really neat part about hiring people with more experience than you is that you get to learn from them.

Always be open about this and thank them when they add to your knowledge. Trust me, you’ll still be adding value for them as well while you learn from them, so don’t worry about that part. I’ve been leading teams in some capacity or the other from extremely early on in my career. And for a good number of years, I was always having to hire people with more experience than me. I absolutely loved learning from them, and I wouldn’t have been as successful if it weren’t for all the things that they taught me.

This post is based on the Ask Me Anything Adil Ajmal very generously hosted for our community on First Round Network. Did you like this format of post? Find it helpful? Let us know by sending a note to [email protected], or tweeting at @firstround. And stay tuned for more interviews like this with other incredible leaders from across the startup ecosystem.

Photos courtesy of Hero Images/Getty Images.