Dan Pupius was the consummate Googler. For six years, he built infrastructure that still supports Maps and Gmail — and won the company's coveted Citizenship Award for exemplary team players. By all accounts, he was the archetype of who succeeds at Google. But he almost never got hired at all.
To this day, Pupius feels that he snuck through their interview process — renowned for being an intensive, multi-stage affair that has since been grafted and replicated throughout tech. While playing his part in recruiting others, he also observed his share of false negatives (people who could have been amazing contributors) and false positives (candidates who checked all the academic and meritocratic boxes but weren't the right fit after all).
So, when he landed at Medium as Head of Engineering in 2012, he wanted to build a recruiting system that wouldn't have the same blind spots. Together with a couple of his colleagues, he set out to build an anti-fragile hiring process that would adapt to new information, wouldn't depend on any one person, and would get stronger over time.
In this exclusive interview, Pupius shares how he did exactly that by applying well-known mechanics of product development. The resulting rubric and hiring machine worked so well that Medium published it online, encouraging other companies to fork it and make it their own. Today, as co-founder of Range — a startup dedicated to cultivating healthy, inclusive and creative organizations — Pupius and his team are forging a new process of their own, ideal for early-stage startups but able to scale. Now you can use what he's learned to transform your approach, too.
APPLYING PRODUCT DEVELOPMENT TO HIRING
When creating a hiring system from scratch, you're actually building two products at the same time: 1) the process itself and 2) the resulting team. Viewed through this lens, every interaction with a candidate must have purpose, yield data you can learn from, and fit into a bigger vision for what you want to accomplish, says Pupius.
Here's what it takes to build truly great products, and how each tenet can apply to hiring:
- Strong intention. If you're a founder, your team will be the product of what you prioritize and reward — and you reward things with your time and attention. Invest the time it takes to build your team as an amazing product (most successful entrepreneurs recommend spending 50% of your time on recruiting).
- Clear and compelling vision. Product development begs the question: What do you want the world to look like? How do you want it to be different from what currently exists? For hiring, ask: What do you want your team to look like? What do you want it to be able to achieve?
- Development principles. You want to have values, beliefs and guidelines that shape how you achieve your vision. For hiring, you should, similarly, have values that will help you make good decisions while eliminating bias.
- Tactics you can test. Product development requires clear, day-to-day actions that execute against your vision, guided by your principles. They're your best guesses for what will yield good results; they're measurable and can be iterated on. In hiring, your tactics are how you conduct interactions with candidates. They should be testable and changeable.
- Measurement that's meaningful. You want to set quantitative goals that will indicate progress toward your vision, and make this data visible to inform your next decisions. In hiring, you want to set metrics and measure whether you're building the team you want. More on this later.
- Organized iteration. The best products have strong feedback loops between users and developers that drive smart changes that have positive impact. Similarly, you want to update your hiring process in intentional sprints based on feedback.
- Designated decision-making. Product development often utilizes the RACI framework to assign one person to be responsible for decision making, one person who is accountable, those who contribute and those who are informed. Similarly, in hiring, you need one person who is ultimately responsible for the final decision on a candidate. Note, this often shouldn’t be the most senior person involved, but rather the person who’s accountable for making sure the hire is successful. You also want to create a disagree-and-commit atmosphere around hiring.
- User-centricity. Just like your goal for any product would be product-market fit, you want your hiring process to fit all candidates — even if they don't get hired. They are your end users, and the best system will be the one that optimizes their experience.
- Automation potential. Once you have product-market fit, your next step is to automate as much as possible. Similarly, once you have a hiring system that produces good results, you want to create the checklists, structures and protocols that will make it repeatable and scalable as you enter rapid growth.
If you apply all of the above to building your hiring system, you'll end up with a stress-tested playbook that you can run again and again. Importantly, you want to document the system you devise somewhere central and shareable. Consider it a living document that can be changed over time as you learn more. It can't just live in your founding teams' heads or on some deserted wiki page — it has to be used and referenced for every single hire.
Now, let's walk through each of the core pieces of a great process in more detail.
VISION
A lot of founders start with weak vision for team-building — which kicks off a weak process. They think it's enough to say they want to build a "world-class team" or "I want us to make all the lists of companies with the best culture." They might be even more detailed: "I want a team of focused, hard workers who fundamentally believe in our mission." But none of this is good enough. Nebulous vision leads to bad strategy.
You need a comprehensive definition of the type of person who will succeed at your company specifically. What is unique to your company and your mission? What unique qualities in people will building that type of company require? Ask yourself, will this vision repel people who aren't a good fit? If yes, that means it's not an empty statement.
Most importantly, vision needs to consider your team as a whole — a single, coherent product.
Just like it doesn't work to build a product as a collection of features, you don't want to build a team that's just a collection of individuals, no matter how exceptional they each are.
Every person on your team has their own attributes and abilities that have to work together — and therefore must be strategically assembled.
To create Range's vision for talent, the founding team took a hike together. "I find that physical activity really helps drive creativity, and being outside the office can help everyone look at themselves more objectively," Pupius says. This extra introspection was very necessary for what came next.
In brutally honest fashion, they listed the traits and qualities that were already on the team, calling out their own and each other's particular strengths and weaknesses. This allowed them to identify what type of people they'd need to complement their shortcomings and extend their capabilities. For example, they realized that they're largely a band of intuitive introverts — who could really benefit from more process-driven extroverts.
Their resulting vision was not a statement, but a list of traits, values and skills that new recruits should have to succeed at Range, buttress the existing team, and maintain balance as more people joined. Now, each recruiting process starts with everyone involved reviewing this list so they're reminded that they're responsible for building a well-rounded, smooth-operating whole.
It’s also advisable to choose one metric that can quickly indicate whether you’re making progress toward your vision. For example, once your company is over a year old, retention makes the most sense. If your process is strong, it’s reasonable to assume people would be good fits, happy with their impact and engaged for the long-haul.
Because they’re still so early, the Range team chose offer-acceptance rate as their top-line metric, since that shows how well they’re screening people earlier on in the process. While you also want to have secondary metrics to hold yourself accountable to the broader vision, having one as your quick temperature-taker is very helpful.
PRINCIPLES
After documenting your vision, think about what rules will get you there. What are the guidelines you want applied to every single candidate? What are the constraints that will force you to think more deeply? Write down what you believe a standard recruiting/interview process might look like, then ask yourself:
- Where might biases get introduced?
- Where and why might bad decisions get made?
- Where might people on the hiring loop have blind spots?
- Where and how might you fail to unearth qualities that would be awesome for your company to have?
Use your answers to these questions to create and enforce a set of principles that will help you find the people your vision describes. Below, Pupius shares what he and his team at Medium (particularly, Head of Recruiting Jeff Lu, Engineering Director Jamie Talbot, and Engineering Manager Jean Hsu) observed while hiring — and the principles they relied on to run a high-quality process. He’s now applying many of the same ideas at Range.
1. All judgments must be tied to observable evidence.
Recruiting is always emotional. But you need to remove as much of that as you can. Within 30 seconds, the average interviewer has already formed a positive or negative impression of a candidate. They either connect immediately with someone they feel like they relate to, or they assume someone they can't relate to is not a fit. This hurts diversity of background and thought.
To prevent this, focus interviews on the list of attributes and skills you've developed as your vision. Then only take observable behavior that either demonstrates them or doesn't into account. What did a candidate actually say and do? How have they behaved in past roles and scenarios?
If you're on a hiring loop and you can't back up a judgment with something the candidate did or said, then it shouldn't count or be shared.
This, more than anything, will hold your team accountable for looking at the right factors. If, during a debrief with your hiring loop, someone shares a gut feeling they have about a candidate, they should be asked specifically to produce evidence to support it. That way, gut feelings that could influence others' thinking, if aired, will get unpacked.
"There have been a few times in the past where I've nearly passed on a candidate because of a perceived character trait, only to be proven completely wrong once they were hired and working,” says Pupius. “Tying all feedback to evidence helps avoid this."
2. Agree on the data you won't be looking at.
A lot of startups borrow Google's emphasis on pedigree. Don't fall into this trap, says Pupius. Instead, hone in on the parts of a person's background that will actually help them do great work at your company. Ask yourself: What attributes aren't actually good predictors for this? What on a person's resume will inspire inferences and assumptions? Take those things off the table.
For example, at Range, Pupius and his co-founders have specified that they won't place any weight on what schools people went to, their grades, previous companies, who they know, etc. Instead, they'll only look for the skills and experiences that will enable candidates to do the work they need done well at the early stage they're operating at.
3. Provide candidates with several success modes.
To field a well-rounded team, you shouldn't offer just one way for candidates to succeed at your process. If you run identical interviews for everyone, you're going to miss out on incredibly talented people who were just not given the right opportunity to shine.
For technical hires, as an example, this means offering each candidate multiple choices to demonstrate their relevant aptitudes: white boarding, a programming exercise, a design exercise, building an app over the course of a couple days and presenting it to the hiring loop, coming into the office to give a tech talk to junior employees, etc.
Think about the skills an excellent hire must have. What are all the ways a candidate could showcase them to you? Let them choose how they want to proceed.
4. Keep team composition top of mind.
Every role you hire for can go in several directions. It's like a multi-pronged fork in the road. "Let's say you need a back-end engineer," says Pupius. "It could go three ways: 1) a relatively autonomous senior engineer who will grow fast, 2) someone with potential to become your chief architect providing technical direction as an IC, or 3) a natural leader who could eventually take over the engineering team. Your choice will open up a whole other set of concerns and possibilities."
To make sure you're always choosing the right fork in the road, you want to have a sense of the team composition you're aiming for at any given time. (As a rule of thumb, it’s also good to know as much as you can about people’s career aspirations so you can understand and anticipate movement within your org.) Here are just a few examples of principles set around composition:
- For every given stage of growth, you want to set a guiding ratio of senior to junior hires. "That way, you won't end up shoe-horning very senior people into roles too early — that aren't right for them — just because they're talented."
- A ratio of generalists versus specialists based on your primary challenges and goals.
- A 1:1 ratio of women and men to keep gender diversity top of mind.
While these composition objectives might shift, they should stay static for multiple quarters to effectively keep the team growing in the right direction. Each of these stakes you put in the ground will help guide which route you choose whenever you have a role to fill. Sometimes they might indicate you don't even need a new hire at all. "Ideally, this principle will make you think more carefully about all the different ways you can get things done."
5. The hiring manager is the decision-making authority.
Seeking consensus on candidates will deadlock and inevitably ruin your hiring process. Pupius recommends creating a principle that will break this up. Designate one person in your process as responsible for making the final decision based on all the evidence presented. It makes sense that this individual be the hiring manager — after all, they'll be working closest with the person.
This isn't enough on its own, though. You also need to prime the rest of the hiring loop to disagree and commit. Just like you'd ask engineering and design teams to rally behind a feature decided on by a product manager, you want anyone who was not unanimously in favor of a candidate to commit to setting them up for success.
TACTICS
The way you set up your recruiting and interview process depends on your own vision and principles. This can vary widely from company to company. For earlier or smaller startups, Pupius recommends just three phases to start:
- Assess
- Validate
- Sell
Before you begin, consider creating a chart that lists all the attributes and skills you're looking for down the left-hand side. Going across the top, have grades: low, medium, high and did not observe (DNO), as well as a comment field to jot down notes on the specific evidence that led to that grade (actual things the candidate said or did in the past). This encourages every interviewer to stay laser-focused on the information that matters, ask questions that will mine this data, and capture their thoughts in real time.
To help with grading, take that list of attributes/values/qualities that make up your vision for your team. In 1-2 sentences, state what each means to you and in the context for your company. Then, under that, give an example of observable behavior a candidate might display to prove they have that attribute. You can group attributes into categories to make this even easier. The Range team has grouped them into buckets labeled Mission/Values, Teamwork, and Stage. Here's just a quick sample:
- Mission/Values: One of several attributes they're looking for is Humanism — i.e. the candidate must believe in the goodness of human beings and that managers should primarily be coaches helping their reports grow and level up (related to the mission of the company). For this attribute, the observable behavior might be them talking about coaching more junior members of their team.
- Teamwork: One of 10+ attributes they're looking for here is Systems Thinking — i.e. the ability to design a product that fits into different aspects of a larger software system. Observable behavior here is a candidate thinking at different levels of abstraction, who has designed software for internal processes within a system before.
- Stage: This bucket is especially important to evaluate for startups. One of three attributes Range looks for here is Comfort with Uncertainty — i.e. whether or not the candidate has executed despite high degrees of uncertainty about a past company or project. The observable behavior is that they've made significant forward progress on something without complete information.
With this rubric in hand, you can initiate your own version of the below phases (this is Range's rough organization):
Assess
- A coffee or casual meeting for the hiring manager and prospective candidate to get acquainted.
- Gauge the person's fit for the company, and particularly the stage they're at if they're venture-backed. Very different people are right for early, growth and late-stage companies.
- Discuss compensation expectations.
- The hiring manager may do an initial backchannel if they know people who know the candidate.
Validate
- 1:1 interviews with people who will work or interact with the role being hired for. These should dig into the attributes, skills and qualities you're looking for using the chart.
- Collaboration time with two members of the team to gauge problem solving and simulate what the actual work experience with each other would be like.
- Skill evaluation (this is where multiple choices are presented to the candidate).
- Deep references.
Sell
- The team gets a chance to pitch the candidate on why the work is interesting.
- You might have investors call candidates to encourage and close them.
- You invite the candidate to join the team for recreation or socializing to get to know each other as whole humans.
If, during the Assess phase, it's clear that the candidate would score low on any of the attributes on your chart, you shouldn't pass them along. But the Validate phase is the most important to get right. Pupius shares a few best practices to make sure you nail it:
Stick with behavioral interviewing. You get great signal from asking people about how they've behaved or handled situations in the past. At Medium, there was a designated career history interview, where they'd walk candidates through their resumes chronologically and ask them similar questions.
One very useful question is, "Who did you report to, and how would that person describe your strengths? Your weaknesses?" If you ask this for each job they've had, you get to see how their strengths evolved. Perhaps they had weaknesses that then became strengths. You get a good sense of learning curve. Also, people aren't as rehearsed to answer the question when phrased this way, they are likely to quote directly from past performance reviews to make sure they're being accurate, and they might end up saying better things about themselves than they would otherwise. Also keep this in mind for reference checking — piece together how someone’s strengths changed over time through conversations with past managers in chronological order.
I'd rather hire someone who knows less but demonstrates a strong learning curve and a growth mindset.
Send your rubric to candidates in advance so they know how they'll be evaluated. Medium started doing this for engineering candidates and saw a very positive impact. "People were coming in feeling prepared, like they'd studied for a test rather than dreading a pop quiz," says Pupius. "This totally changed the tenor of conversation and their comfort level. You don't get anything out of people feeling nervous and on edge." It's also a very positive signal of how transparent your culture is internally.
Set up shadowing to calibrate new interviewers fast. None of this will work if your interviewers don't share a mindset about what they're looking for and how they're asking questions. Making sure this happens eliminates variables, so that candidates have an equal playing field.
Medium approached this efficiently: Before letting people do interviews on their own, they'd have to shadow more experienced colleagues twice. Then they'd do two reverse shadows where they'd lead, but the colleague would watch (in a non-awkward way of course). Afterward, they'd have a candid discussion, with the more seasoned interviewer offering feedback and explaining why they do things a certain way. If you have a young company, start people shadowing right away to build your corps of potential interviewers.
MEASUREMENT
Remember, the "end user" for the hiring process you're building is the candidate. So, while retention may be your top-line metric for all of this — the more immediate measurement to track is candidate experience.
Pupius recommends instrumenting your hiring funnel as soon as possible. Companies like Lever and Greenhouse make this easy now. But the important things to track right off the bat are:
- How many candidates are coming into your funnel and from where?
- Where are they dropping out before accepting or declining an offer? What reasons do they give?
- How fast are people moving through? What has held them up if anything?
- What's your close rate? When people decline, why?
- Are there any demographic trends?
Every week, you want your hiring managers to review these analytics and hypothesize a) what's going on and b) how to make things better. Early on, the founders should be involved in these meetings. In particular, the CEO should own hiring metrics for as long as possible. Later, the hiring manager should convene with your recruiting leader and the head of their functional area — who should eventually take ownership of metrics for the team.
At Medium, there was a point when they noticed almost all candidates entering the funnel were referrals. They wondered why there wasn't much inbound, reconsidered the places where they were sourcing, and fixed the issue. You won't be able to make these types of high-leverage changes without regular review.
Most importantly, Pupius' team at Medium would follow up with candidates — both those who got offers and those who didn't — to get their feedback.
"You want all candidates who didn't get the job to still have an incredibly positive impression of your company and your process," he says. "The world is small. Reputations are long. You want them to have wished they would have gotten the job — so much so that they'll still tell their friends to apply."
They asked all candidates (hired and not) variations on the following list of questions — designed to test how user-centric the hiring process felt:
- What was your overall experience like working with the recruiter/hiring manager?
- Were the questions you were asked good for testing your skills as they related to the role?
- Did you have any questions about Medium that didn't get answered?
- Did you feel like the process and culture were inclusive?
- Do you feel like our process assessed you fairly?
- What questions do you wish you were asked?
- Did interviewers make you feel comfortable and help you through the process?
Feedback they collected through these surveys led to them tweaking questions, candidate options, and more. It also pointed to who the strongest and weakest interviewers on the team were over time — so they could turn the best interviewers into coaches for everyone else.
One thing Pupius saw consistently was how continuing to be user-centric through the entire process made the biggest difference. Sometimes, hiring managers would follow up with non-selected candidates afterward to say, “Hey, I'm happy to connect you with other terrific people who are hiring, who I respect.” This got tremendously positive results, impacting their perception of the entire experience.
Because retention takes time to measure in a meaningful way, you can check in with that quarterly and annually to make sure the people who do make it through feel similarly happy and satisfied with outcomes. Pupius also recommends quarterly pulse surveys of staff with an NPS component — would people recommend working at your company to a friend?
ITERATION
The final rubric Medium published was the result of a lot of iteration based on measurement — a herculean effort made by Medium’s Director of Engineering Jamie Talbot. For instance, a lot of the immediate learnings pointed to the need for more interviewer calibration.
To get the most out of iteration, Pupius recommends batching feedback and approaching it in sprints. To be systematic at Medium, they'd run quarterly retrospectives on the feedback gathered from all candidates. First, they'd look at which interviewers were newly calibrated to understand performance ratings. Then, they'd decide as a group what actually needed to be changed to create a better experience and move numbers in the right direction.
Instead of approaching changes as a hodgepodge backlog, they'd prioritize them into themed batches. For instance, if inbound interest was the most pressing issue, they'd tackle all action items around that at the same time. The next quarter, they might take on close rates, etc. It was helpful to iterate around one theme at a time so they could attribute new data and feedback to specific changes made. Pupius and his team found this yielded more learning per quarter. And having light constraints allowed them to achieve more of their goals (he’s a fan of the Theory of Constraints for a reason).
Ultimately, this is what you should be maximizing for early on: learning per unit time. Startups have a window of opportunity to perfect their hiring process before they hit the prime time of rapid growth. Use this window wisely by not just figuring out what works, but fine-tuning and documenting it in a simple, straightforward way. Like everything you're doing, the outcome is subject to change — but it's shocking what a difference it can make. If you want a winning product — build your team like one.
Thanks to Jeff Lu, former Head of Recruiting at Medium (now at Reddit), and Jean Hsu, former Medium Engineering Manager (and now leadership coach), for contributing their recollections and tactics.