Why everything we’ve been taught about quitting is wrong — Annie Duke

Jessi:

Okay. Can we start recording? Perfect. So I was kind of curious just to start off by sort of what your impressions were when you first started working with startup founders and investors as a consultant. Was there anything that was surprising to you, or did you see a lot of connections and parallels either between your research or your experiences as a poker player?

Annie:

That's a broad question.

Jessi:

Yeah, just broad to start it off.

Annie:

Yeah, that's super broad. I mean, I don't know how negative I'm supposed to go here, but...

Jessi:

No, totally.

Annie:

The thing that when I first started doing this work was really surprising to me but now is no longer surprising, but still kind of surprising, even though I know it, I'm still surprised is that how much people are willing to kind of go with their gut or at least to not make explicit what their process is for entering into a decision. So I remember having a conversation with a group of partners that were not First Round partners, where they told me basically they know a good founder when they see one, and I was like, "Okay. So how do you sort of decide? What do you think about the..." No, I just know, and I think that for most of these things, and obviously when you're dealing with the startup world, you're talking about things that have a lot of uncertainty and when there's a lot of uncertainty, I think that people tend toward sort of saying, "Well, I just go with my gut. I just know what a good decision is."

I think in part because they kind of know that if you were to ask them to get you a specific answer, they would not know the answer because the answer is kind of unknowable, but just because the answer's unknowable, doesn't mean that you don't know lots of things, nor does it mean that making explicit what's going into your decision isn't worthwhile both from the sense of increasing decision accuracy, but also from the sense of being able to get feedback. So I'm going to try to ground this because I know that that's sort of lofty. I mean, if you want clarification, let me know.

So from First Round's perspective, as an example, we were talking about trying to do some evaluations of investment quality over the course of the time before... This was when I first started working with First Round and my first thing was, "Well, can I see the voting data?" And of course for me, I'm thinking voting data is what do you think of certain pieces of the puzzle. If you're talking about, "I know a good founder when I see one," did you write down what you thought of the founder? Did you think about what are the qualities of a founder that you believe are predictive of future success? So this is what I asked for and it was like, "Well, I don't have that," and I was like, "What?" And it was like, "Well, we kind of used to do it, but of course, it was very sort of high level and mushy, but then we stopped doing it and now we just take the founder pitch and we vote." The thing about it is I think that's much more the norm than you would think, but you can sort of see the problem here.

I mean, separate from you ought to be able to be explicit about what you think. I'm sorry. So let me try to divide this into two pieces because I think this is really important. So there's the piece of what is it doing for your decision quality in the moment, and then... So actually let me talk about three pieces. What's it doing for your decision quality in the moment, what's it doing in terms of knowledge transfer among the group, but then also what is it doing for you in terms of feedback? So let's start with feedback.

Because they were asking me to evaluate decision quality retrospectively, I now didn't have the data to be able to do that. So I had some proxies, right? I could look at performance compared to a benchmark, or if we went and made a definition of what makes a good company, I could sort of look retrospectively and say, "How good are they at sort of picking those things out?" But in terms of the actual opinions of the partnership itself at the time that the company came in, I couldn't now go back and look at that. That creates a problem where I now have to sort of collect data around that. That make sense?

And obviously, what you'd like to do is to be able to say, "When you give an opinion of whether you should think you should invest or not, what are the things that you're thinking about and how predictive is that opinion of future success, so that I can now feed that back to you both in terms of understanding how good is this partnership at those questions, but also here's where you can get better, and let's think about how to improve that," right? So that's kind of the idea of creating these feedback loops. So that's kind of piece number one that you're losing there, which is a very big thing.

And I remember saying at the time why we need to start doing this, and just so you know, we're not going to be able to close the feedback loops for a couple years or more because obviously you have to wait for companies to cycle through as you start to get results when you're investing at seed. But I said, "You're going to have to wait around, but then you're going to be happy that you did it." So the partner meeting that I just came from was actually the first feedback session on voting. So it's like, "Okay, we're finally there. It took a while." Okay, so that that's kind of thing number one.

So thing number two has to do with decision quality itself. So the problem is that when we're making decisions, we know from all this working cognitive science stuff, our decisions are super noisy and that they tend to be super biased. So what I mean by that is that when we're thinking about a company or a founder or someone we want to hire, or a strategic choice from a product, like what features are we going to develop, what are we not going to develop, where are we going to spend their time? There's all these questions and it's all allocation of resources, whether it's money or time or effort or whatever. On the bias side, we're riddled with sort of the way that our mind works such that those judgements aren't particularly good. So when we're making a decision that's a prediction of the future and those predictions tend to be pretty poor because there's overconfidence, there's confirmation bias, there's availability bias. I'm happy to define all those for you. Illusion of control, all these things. And not only just those things, but we're all kind of handcuffed to our own personal experiences, right?

So for example, if you were just sort of looking into the eye of a founder and investing and you were successful, you're going to tend to think that that's a good way to invest, but you don't really have a counterfactual to compare it to, which is, "Well, what if I were more systematic about the way that I approach this problem? Would I do even better than I've already been doing?" I mean, we don't have those things to compare to, but someone who's been doing it that way and has success, that's like their model of the world, particularly if we sort of have developed some sense of subject matter expertise or we've been doing it for a long time, and doing pretty well, these create very strong models of the world and we kind of get trapped in those models of the world so that when new information comes in, we'll tend to interpret it to fit our model.

So as an example, a company does really well and you're kind of looking back at that and you'll be like, "Oh, I remember. I thought the founder was great," but it's unclear whether you did because you didn't actually record the opinion, that kind of goes back to the feedback thin, or you'll remember one company where you thought the founder was amazing and you invested in them and it worked out, but you forget the companies where you thought the founder was incredible also and it didn't work out or where it's kind like the founder didn't have a lot of personality, but the company was amazing, right? This is kind of the problem when we're not systematic is that when we're looking at the world through kind of just that gut or our own point of view, or just we had a discussion and now we're going to vote, which is basically using your gut, that's where all of that bias lives, that's where all of that sort of degrading of accuracy happens.

So the way to kind of help with that is kind of twofold. The first is to think outside of a particular decision that you're facing. In other words, I'm not in front of a founder, but I'm thinking in general about the decision space that I'm thinking about. What are the things that I think are judgments that would help? So for example, when you're investing, do you care about the market? Do you care about the team? Do you care about the product? And so we know that these are things that you care about, so you want to sort of think about those broad buckets. So First Round was doing a little bit of that I think a couple years before I came in, although they had stopped, but then they did take it a step further, which is to say, "If I'm going to think about the team, what specifically are the features of a great team?"

So some of the ones that we use, which obviously, you may not want to put into the thing, but I'll just say to you are storytelling, like how deep do they go on the founder, on the problem? What is their understanding of the space that they're in? Whatever, so and so forth. I think there's four that we use or five for team rather, and then you think about the same thing for product, like what is the acuteness of the need? Essentially, is this product a big painkiller, right? How strong is the execution of the product? So and so forth. So we can think about that, and then on market you can think about what's the total TAM, how crowded is the space, how strong is the moat that you can create, are there headwinds or tailwinds? So and so forth. So we can think about what those specific judgements are, and then we can... Now we've done that when we don't have a company in front of us.

So what we're saying is whenever a company comes in, I've already sort of decided what are the things that I need to make judgments about, and I'm going to record those judgments for posterity because I know that people are going to come back and look at it, and then that helps to overcome bias and it helps to overcome noise. Oh, I forgot to define for you what noise is. Sorry. Noise is different people have different judgements, but also you'll have different judgements. So if a founder comes in and pitches on one day, you may think about them differently than if they come and pitch two weeks later, and it would be the identical pitch, but you'll judge it differently depending, and I'm sure we've all felt that before ourselves.

Jessi:

Or the studies about how a judge will be harsh if they haven't had lunch yet, or whatever.

Annie:

Right. So those are all forms of noise. So when we use this kind of process where we think in advance what specifically are the judgments that we need to make, this helps to reduce noise. It also helps to reduce bias because it's almost checklisting. You can't ignore a bad signal or ignore a good signal. You have to rate every single one, which helps you sort of ground yourself in reality. Aa good example is when you're hiring for a job that doesn't require a lot of charisma, the charisma of the candidate will cause you to rate the other things, to think about them more highly, even though it doesn't really matter. So if you think in advance like, "What do I want? I'm hiring an engineer. What are the things that I'd like it to hire them well?" It helps you to sort of get over that. I really want to invest, so I'm going to highlight certain things and lowlight certain things because you have to explicitly rate every single one. So that's sort of the first piece.

But now the second piece is that... So that's kind of thinking in advance, helping yourself in the future to be a better judge by thinking about it now before you actually have to face the decision, and disciplining the judgment so you're making the same judgments across every single company that comes in because you decided in advance what matters. So then the second way that you can really discipline this bias problem is to have other people make the same judgements as you and compare, but that only works if they do it independently. So if you were to say, "All right, let's all talk about market," and then you just start talking about market as a team, that's going to produce very bad judgment because there's going to be all sorts of cross influence and people are going to try to convince each other. Sometimes it can become combative, which is bad for decision-making and there's going to be contagion. My opinion is going to be a contagion.

So if instead we say, "Okay, we've decided in advance. Here are the things that matter to this decision. Here are the things that we have to judge. So we have this broad bucket market, we have these detailed criteria, mediating judgements that we're making that help to inform those judgements," but now I fill it out, you fill it out, Bill fills it out, josh fills it out after seeing the founder without speaking. Then we can see everybody's opinion and then we can discuss the places where those opinions differ because we don't really care where they're the same, that's [inaudible 00:14:30], but we don't need talk about, so we talk about the places where it differs before we actually make the final decision.

So notice now this solves the first problem, which is the feedback problem, because now everybody is doing this and we can check back in on those votes across the partnership, but then also for each individual, they're being disciplined in two ways in terms of what their own biases are such that the forum itself is de-biasing them, but then the decision that gets produced by the group will be less biased because you're actually seeing everybody's opinion independently and being able to maximize the amount of dispersion that you actually get a view into. Does that make sense? I feel like I'm boring you.

Jessi:

No, no. This is perfect. So my question on the discipline and the hygiene side, from the founder perspective, obviously there's the uncertainty problems and the bias problems you pointed out, but one thing that often happens is, especially early stage, they're like, "We don't need any process," or, "We're scrappy. We just kind of do things by the seat of our pants," and then when they scale, they realize, "Oh, crap. We have to go back and do that." So I guess sort of what is your thought on what a good decision hygiene or process looks like early stage and then what needs to change as you need to scale? What should a super early founder be doing that's not overkill, but is appropriate for their stage?

Annie:

So, I personally don't think from a decision process standpoint that it should change much. So I think there's ways that where founders go wrong, which are the opposite ways of where enterprises go wrong, but neither of them is getting it really right.

Jessi:

What's an example of a difference or a way a startup goes wrong in your experience?

Annie:

I'm giving an example from an enterprise to start.

Jessi:

Okay.

Annie:

They'll have some decision where... Well, first of all, they have layers of approval, which isn't great. Most things have to be by consensus, they have to be by the book and they're super slow on decisions where they should be fast. That's a place where they could learn from startups. So what starts to happen in an enterprise is once you get to the enterprise state, you have built a status quo. This is the way that we do things, these are the products that we're exploiting. Exploiting not in a bad sense, in the sense of this is what we're making and selling and we're going to keep making and selling it. And people start to develop career risk, and so they want approvals and coverage and they want to do things the way that it's expected, and then also people aren't authorized to do their own work because within the context of a huge enterprise, a whole bunch of things go wrong.

So I think enterprises are not innovative enough and they move too slow on a lot of stuff, but that doesn't mean that for... A startup has the opposite problem, which is that they don't have enough process in place because it's really easy to put this kind of process in place. If you're thinking about... I'm all for just do that stuff and have the world tell you what to do. I think that's really good, particularly when the cost of doing so is really low and the decision is really reversible. So we can think about MVP as a really good example of just do it fast and just find out what the customer thinks, find out if it's buggy, find out if it works the way we think it's going to work, and then we'll fix it after that. But what product are you going to develop? Now, that's another question. So once you've gotten to like, "Okay, I know we're going to develop this particular product," then sure, create minimum viable products, send it out, see what happens, blah, blah, blah, feature development, right?

For some of those things, it's fine just to be like, "Let's just try it," but for other things, when you have limited resources, you ought to step back and have some regular process for triaging. Okay, we have this whole list of things on the product roadmap. I mean, I don't know. It feels like a lot of startups don't have a product roadmap. It would be helpful, by the way. Even if you're going to move fast, you want to know what road you're driving on and where the exit signs are, right? You want to think about there's a difference between are we going fast in sort of the execution and trying some things out, but are we also under standing directionally where we're supposed to go?

And that's something that people should be stepping back on, and it's the same thing with hiring people. You shouldn't hire someone you met in a coffee shop, right? You should be like, "This person has been one of seven. We should be thinking in advance about what challenges that we expect this person to be addressing both short-term and long-term. What do we think about for us, what our values are as a team, how is this person going to fit in with those values? Do we see this person growing with the job? Can we imagine this person as leadership in a company going forward?" Know what it is that you're trying to bring onto your team and then have people talk to the person and just fill out a little form, ranking the candidate that you're talking to on those features that you've decided in advance are important to you because yeah, I agree. You should move fast when you're just sort of trying to put a product out, but you shouldn't move fast necessarily when you're going from six employees to seven.

That's going to be a huge chunk of your team, so just take a step back and say, "What is it that we value? What are the challenges that we're expecting this person to address? What are the features of someone who's going to work well in this environment? How are we going to know that through an interview question, whether they're down with being one of seven, where it's like roll up your sleeves and just do a whole bunch of crap that you... You might not even have a job description." Okay. One of the questions is, is this person going to be comfortable not having a job description? Is this person a self-starter? Do they know when to slow down and have a discussion about what they're about to do and when they can just go ahead with no permission whatsoever? I don't know what those questions are, but figure it out for your team and to take a little time in advance. It will pay off.

And same thing with yes, go fast when you're creating a minimal viable product. Yeah, but which MVP are you going to create? Take a little time to say, "Here are our options, here are the different things we can do," and what you can do is just... It's the simplest thing, this doesn't have to take a lot of time. Just have the people on your team, maybe there's three of you. I don't know. Have each of you force rank them. If you were king of the world, what would you do first, what would you do second, what would you do third? Or we have two different directional options, we can't do both of them because we're too small, and we could go B2B or we could go B2C. Okay, figure out what is the feedback that we need from people in order to have a coherent discussion about whether we should be B2B or whether we should be B2C.

Let's elicit those judgements in advance independently so that we know what we're... And then we go into the meeting and we can then have a discussion about the places where the three of us actually have some dispersion so that we can understand what the decision space looks like, and that's going to pay off and it's going to build a muscle about sort of how do you navigate these types of decisions, how do you triage, when is it okay to sort of be fast and errorful, and when do we actually want to take more time, what are the high impact decisions where it's hard to a reverse, and it's really going to matter to us and what our future is? Build that muscle, so that as you start to scale, you can bring that muscle into that scale with you, and that should actually help you to lose some of the horrible kind of slow down and layering that comes from reaching sort of enterprise level.

Jessi:

That's what I was going to ask about is the what... Because I feel like we see this more with folks who are series A, B, C. They're like, "We're not as fast as we used to be. We've slowed down. I really want the team to move faster." What sort of the advice you have for folks who are like, "I need to speed up my process. We're going too slow."?

Annie:

Yeah. So it's always this thing you sort of forget to understand this question of what is the impact and what is the reversibility of the decision. So if I get this wrong, really how big is the impact on the org and how reversible is the decision? It's the difference between hiring an intern and hiring a C level exec. If you're hiring a CFO, you should probably go slow. If you're hiring an intern, you should go fast. You should give your engineers lots of free time to work on problems where who knows what's going to come out of it, right?

It just gives them the free time to be able to do that. If they want to spend the time debugging, fine. If they want to spend the time trying to create a toy feature that might turn into something, cool. It's not going to have an impact if they're just doing these small things that isn't being added into the code. But then you can also start to think about, "Well, is there a way..." And this is what you're doing as a founder all the time, but you lose when you get into a much bigger org, "Is there a way for me to make this a smaller decision than it feels like?"

So this happens all the time with a product roadmap, for example, is that you feel like you have to get it perfect instead of saying, "Well, here are three ideas that we have. We're now bigger. We don't just have three or six people on the team. Can we actually get a group kind of working and testing and starting to think about how they might code this, or even starting to code this and creating something? And someone else could sort of code this and then we can kind of figure out which is the best path to pursue having gotten more information about it, but we don't need to decide perfectly in advance. We know that this is the right path." Well, I think that you lose a little bit of that kind of testing mentality, which is really good, and you feel like you have to get it just right from the start when you're bigger.

Jessi:

And that's why you spend two years working on a feature no one wants, and all that.

Annie:

Right, exactly. And the answer is you can't. I don't care if you spend two years on it, you're still not going to get it just right, so you shouldn't lose that mentality of let people just sort start and try it, or see where it goes and what it feels like and what it looks like, test it with a small group of customers, let the engineers have some free time. You have to keep realizing that in some ways you have control over how big the decision is. So if you approach the product as if I'm developing a new feature that I have to get it perfect because I'm going to roll it out to everybody, as opposed to, well, why don't we just try to create... What's the smallest version of this that we can do at the least cost to us? And it doesn't matter if I have 300 employees or 600 employees, or six. I should still bring that mentality with me.

Jessi:

Mm-hmm (affirmative) Yeah. No, that makes sense. So we've already kind of done one, but I would love to kind of run through some of the examples of decisions or areas that founders face. We've already talked about hiring a little bit, but just kind of anything that comes to mind for you in terms of a bias that's particularly salient here or a tool or mental model or something that might be helpful for folks as they're kind of doing it. So just can be kind of maybe a list or a section of that kind of sort. So one is sort of in this working with customers area of particularly early stage.

We have a lot of founders who are co-designing with early customers, design partners, they're getting lots of feedback, it's going in different directions, and they don't really know how to find a signal in that, or make a decision on which direction they should go, or they bring their own biases into what they want to build. So kind of do you have advice for folks who are in that early building stage and getting feedback stage for customers of either how to pick among conflicting advice and directions, or how to avoid their own bias? Anything that kind of stands out to you there?

Annie:

I've read an interesting paper recently that when you're doing at least customer interviews, as soon as there's less than 5% of the information is new that they tell you, you should stop talking to them.

Jessi:

Oh, really?

Annie:

Yeah. So how many customers are you supposed to talk to or potential customers are you supposed to talk to. You stop after you have an interview where less than 5% of the information is new. It was an interesting paper I sent around to the partnership. Okay, so that's a really interesting problem. So the first thing is to understand that there's no decision that you're going to make that's at all hard, that's going to be consensus. So don't feel like if you're co-designing with customers that all of the customers have to agree on what you're supposed to be doing. That would be really unlikely, and you don't necessarily need to agree with them. You should be the thinking about that is feedback and partnership, thought partnership, in order to help you to understand where you're supposed to take the product, but part of the reason that you're doing what you're doing is because probably what you're thinking you need to do is not obvious because if it was obvious, then other people would be doing it, maybe some of your customers, so just remember that. Really important to remember.

I mean, the thing about this kind of thing is it sort of rinse and repeat, right? So you have this thought partnership, but you're thinking about different directions that you want to go is to actually think very clearly about what is the feedback that you're trying to get from the people who you're asking to help you. Things like understanding for the customer problem acuteness, right? So you can have the customers rate that on a scale of one to seven. Have them give you an answer in a precise way where you can actually see dispersion across the people that you're asking, and then ask them to give you a rationale for why the problem is so acute.

Then you could ask them like, "How well would this feature reduce the pain of the problem that you're having? On a scale of one to seven, where seven is it would be amazing and totally reduce my pain, and one is it wouldn't address it at all, and three is pretty good." You are listening to these kinds of judgments when you're having free form conversations with your customers. That's what you're doing when you're like, "Well, what do you think about this? Would this solve this problem? What problems do you think it would address? How bad are the problems? What would you like to see?" These are all things that you're getting sort of in the conversation, but if you have those as a free form conversation, you're going to learn less than if you start off by getting their opinions in advance.

So if you get their opinions in advance, one thing is you can see how much dispersion is there among the customers, and also what's helpful is you'll avoid going in knowing that the person that you just talked to wants you to go in one direction and you're then going to have a bias probably toward the last conversation and that you had, where you're going to bring that bias into the next conversation that you have, which is going to be you saying, "Well, what do you think of this direction? Because I was just talking to somebody and they thought..." And now you haven't gotten their independent thoughts. So if you can get those judgements and then see, "Okay, where is all the dispersion?" And then when you're talking to customers, you can focus on the areas where you know that they disagree and you can start exploring where that difference of opinion is coming from.

So if you know, for example, that people really differ on particular feature that you're thinking about developing. I'm just making the up. I just want to say that because obviously it's not a perfect example, but then you can go and you can say, if you know that you can say, "Well, I really want to dig into why you're so negative on this feature," and then they can tell you and then you can say... Because you'll have rationales, you'll say, "Well, a few other folks have said X, Y, and Z. What do you think about those thoughts? Does that resonate with you at all?" It directs the conversation in a way that's really helpful, and then also because you've gotten those opinions independently, you will see that there's a lot of dispersion in the way that people think about it.

Now, if it turns out that everybody thinks that's a super acute problem and that what you're thinking about developing is perfect for them, and if it worked they would definitely buy it, that's good signal too, but then it's obvious what you're supposed to do, right? But we're talking about non-obvious things. So once you see that there's dispersion, what you realize is that there is no way to actually come to consensus because the whole point is, first of all, people have their own needs, and second of all, sometimes people don't know they want something until they actually have it. I didn't know that I wanted an iPhone for most of my life.

Jessi:

Yeah.

Annie:

I didn't think that was a thing that I needed, and then I did need it and now if I lost it, I would have a panic attack.

Jessi:

Yeah, exactly.

Annie:

So someone needed to show me this was something I needed. So it allows you to kind of get past that idea of, "I need to come to some sort of certainty," and having been informed by that, you need to realize the decision is your own, right? But obviously if every single customer said, "No way, no how. That's stupid." I don't think I would develop it. But if it's somewhere in between and you've had a good conversation with them, you should feel free to go ahead without consensus.

Jessi:

Okay. Yeah. No, that's super helpful.

Annie:

I don't know if it is. You can tell me if this doesn't make any sense [crosstalk 00:33:44].

Jessi:

No, it definitely does. On the hiring front, so we talked a little bit about adding more rigor to your process in terms of like figuring out what you want and designing your questions around that. Are there specific interview questions that you like to ask or think more founders should be asking when they're hiring people or maybe questions around assessing decision making specifically? Is there any recommendations you have for founders who are like, "I don't know what to ask people in interviews."?

Annie:

Well, okay. So the first thing is that it does matter to take the time to figure out who it is that you're trying to hire. So I do think that that's really important because if you understand those features, then that's going to direct the way that you talk to them, but particularly in a startup world, the questions that I would be asking are about how do you understand uncertainty? When you're thinking about something, how much data do you need to collect before you're willing to act? Can you give me an example of a decision you made where there was lots of disagreement and how did you actually manage to come to a decision given that so many people disagreed with them? And then I think that one of the things that I love as a question is to ask some of these kind of what are called [inaudible 00:35:16] or back-of-the-envelope questions.

So a classic one, but you should come up with your own, would be... They used to ask this, it's a silly question now because it's not something that people really do anymore, but a question that people used to ask is how many piano tuners do you think are in Chicago? So the reason why this type of question is so good is because this is the problem that you have all the time in a startup, which is of course a reasonable answer is, "I don't know," but you would not want to hire someone who stopped there because you're only deciding about things where a reasonable answer is I don't know. The question is how do you think through the problem? How do you figure out what are the pieces of the puzzle that I need to know in order to actually get to an answer here?

So in the case of the piano tunes in Chicago, again, you shouldn't ask this question because it's a kind of a silly question at this point, but you would say, "Well, let me think," right? How many people live in Chicago, you would take a guess at that, then you would say, "Okay, so how many people of those live in the same household?" That would give you an idea of the number of households in Chicago? Then you could imagine, "Well, if I think about what percentage of those people do I think have pianos," and obviously you're estimating all of this stuff so you're thinking, "I have a piano, but I have five other friends who don't have pianos," and you kind of sort of come up with an estimate of how many households do you think have pianos percentage-wise.

So let's say that you think that 10% of households have pianos and there, I don't know how big Chicago is, there's maybe 4 million people in Chicago. I'm taking a guess. The average household is probably given single people and whatnot, maybe the average household has three or four people in it. So maybe there's a million households or a million and a half households, let's say a million and a half households in Chicago, and I think 10% of them have pianos. So there would be 150,000-ish pianos. And then how many pianos could one piano tuner service? That would be the last part of the question. So the idea is not to know that did you get that it was 4 million people in Chicago right. I have no idea whether that's right. Okay, I have no idea. The point is did somebody understand how to think through the problem and were they comfortable thinking through the problem when there wasn't a correct answer?

Jessi:

Yeah, totally. Another question I have and it's not necessarily a decision, so feel free to say pass, but given just how competitive the talent market is, our founders are just struggling a lot with hiring and convincing candidates to join and take their offer over another offer. Are there any tools that are useful in that kind of scenario to influence a candidate's decision to join your startup, or?

Annie:

Yeah, interesting. Don't ask them about the piano tuner. I was thinking good questions to ask, are you willing to never see your family? Because that's part of the thing when you're thinking about the features of someone in a startup. You should ask actually direct, "Are you going to be okay with the fact that you're kind of not have a life for a while? Why do you want to do this? Do you care about this problem, or do you just want to be part of the startup environment?" You should get someone who really cares about this particular problem and is excited to dig into that problem. In terms of convincing people outside of just the package, it depends a little bit on the person that you're hiring. So there are some people who have the mindset where all they're going to do is look at the compensation package and unless you're willing to pay more and give more equity, it's not going to matter much what you say because they're very transactional on that way, and more power to them.

Takes every kind of person to make the world great. So that's definitely something that you have to understand is that some people are just going to be looking at the compensation package and there's nothing you can do there, but that's not true of everybody. So obviously if you have a nice team, that's helpful because maybe they'll think that that's really cool if you have a product that, or if your startup is something that they really care about and are passionate about, then you should be able to convince them away from other startups, but again, that you don't have much control over. So the question is what do you have control over? And what you have control over is getting them endowed to you. So what does that mean? You want them to feel some ownership in your company. I'm not talking about equity ownership, I'm talking about ownership of the idea or the process.

So it's helpful to get them to sort of give you advice is what I would say. You can interview them and then you can say, "Hey, it's really interesting. The market is really hot, and obviously you're in the market, so I'm interested in your advice. What do you think that founders can do in order to attract top talent?" [crosstalk 00:40:46], and you're saying, "I don't know what's going to happen with us, but obviously everybody's facing this and you're uniquely qualified to give me some insight into this," and those kinds of things are actually more helpful than you might imagine.

Jessi:

Yeah, I like that.

Annie:

So bring them into the fold and the problem that you're having.

Jessi:

Yeah. So the other end of this is the firing decision and then the common advice is to-

Annie:

First time you think about it.

Jessi:

Hire slow, fire fast, but people don't actually tend to do that in practice. So I guess how would you advise someone who's working through that decision of whether to manage performance or just make the quick decision to fire someone?

Annie:

Yeah, okay. So in an ideal world you would just fire them, but we don't live in an ideal world. So hiring is very, very uncertain. The best hiring processes can get you to somewhere around 60% accuracy, maybe 65 if it's like... But startups are not going to be that environment. So you have to understand that there's going to be a lot of fails here and I hate to call them even fails. It's like it's just sort of expected that that's going to happen. The problem is that... I think Richard Thaler, Nobel laureate and he said this really well, "we don't like to close an account in the losses." I think this is a really good sentence from Richard Thaler. We don't like to close an account in the losses because this goes not just for hiring, but for features that you might be developing or a particular direction your product is going in, or the startup itself.

What that means is that... So you can think about it like when we start down a path, that opens an account in the same way that if you buy a stock, now you own that stock, right? If you start developing a particular feature, it's like you own a stock, it's opened up an account, and now that stock can go up or down in value, but so can that account. So you'll hire someone new, and if they're working out that account is going up in value and if they're not working out, it's going down in value. Does that make sense? I just want to make sure that it makes sense. So the problem is that we as human beings don't like to close accounts in the losses because when you close an account in a loss, there's lots of implications that you can't ever get it back. And I'll tell you the problem with the thinking in a second and then how to solve for it.

So what that means is that if I hire someone and they're not working out and then I fire them, then I have to just take the loss, whereas if I keep them on and say, "Oh, I can just manage them and make them better," then maybe I can make that account go back up in value and I could get out of this losing position that I'm in. So as long as I keep you on, maybe I could turn it around, right? Once I do, then I've lost. I've wasted the time or the effort, or I made a mistake in hiring you. That's the moment that you have to turn it into a failure. So you can think about the difference between failing and having failed, and we do not like to move from that failing to the having failed moment.

So that's what he means by we don't like to close an account in the losses is that you sort of have a loss on paper, things aren't going well, but it's not until you actually quit it and firing is a form of quitting that now it becomes, "I have failed. Now it is done," and you have no chance of getting it back. And that's separate apart from you have pride of ownership of the decision. You make the decision, so if you have to fire them, what does that mean about your decision-making capability? Which we don't like that piece either. There's a whole bunch of stuff that we don't like.

So there's two ways that you can solve this. The first is kind of a framing issue, which is to say, "What could I be doing with the money and the equity that I am paying this person and the time that I am putting into this person? So if I free that money up and I free that equity up and I free that time up, what could I be using that for?" So the reason why that's just kind of a helpful frame to think about is that it allows you to start recognizing the opportunity cost that you're incurring separate and apart from the cost of having an employee who just happens not to be performing, it's costing you the opportunity to hire someone else who would be.

So when you start to think specific about what are the things that I could do with those dollars and the time that I have to manage this person and the equity, then it starts to focus you on the opportunity costs. So it's just kind of like a helpful thing for you to ask yourself is to reframe it not as like, "Could I manage through it, or could I make them perform better?" But, "If I freed that up, what would I do with it?" It's actually super helpful. Okay, so that's kind of piece number one.

The second piece has to do with setting for yourself what are the criteria under which I would let this person go? Now, here's the thing: You would like to do that in the beginning, but you don't. Okay, so the first time that you have the thought, "Oh, I don't think this person is working out," you're probably not going to let them go. Human nature. Fine. Okay. So I'm going to accept that from you, but what you have to do at that moment is to say, "What do I need to see over the next X period of time for me to feel like I've regained confidence in them? What are the benchmarks that I think they need hit? What do I need to see in terms of performance?" And actually cast yourself into the future.

So this goes back to when you're in it, you're not going to make a particularly good decision. When you're actually facing the decision down, you're not going to make a particularly good decision. So if you can get yourself to sort of get away from the decision now and think about it as something you might do in the future, a future version of you might do, it's actually super helpful to getting you to actually do the thing that you ought to do. So this is something that you should actually [inaudible 00:47:33] with the employee is to sit down and say, "Okay, let's kind of talk about what are the KPIs that I expect to see in two months, or in one month, or in three months? What is the work product that I expect you to see? What is the level of effort that I expect from you? What is the precision and quality of the work that I expect to see from you in the next period of time?"

Those now create benchmarks for you, and then what happens is that when they don't meet them, particularly because you have made them clear to the person that you're thinking about, because you've done it, you've shown them it and you could actually have them write it too. Here are the things that I ought to be producing for you. Then when that actually hits, then you can actually let it go because it basically creates what's called a pre-commitment contract, which is you're committing in advance to certain actions in the future. So all across sort of the behavioral science world, pre-commitment contracts are one of the best ways to kind of improve decisions and outcomes, and it's basically I'm thinking now about what I ought to do in the future and that's going to improve my ability to do it.

So a very simple example would be having a designated driver or making sure you don't drive your car to a place where you're going to be drinking, and instead Uber there so that you don't have a way to drive home because as much as you know right now, "After a couple of drinks, it would be stupid for me to get into my car," we all know that when you're faced with that decision and you're like, "Oh, should I leave my car here?" We're all very good at convincing ourselves. So instead we're like, "Okay, I'm just going to make it so that I know I'll have to take an Uber." So that's essentially a little bit the flavor of what this is doing for you.

So you can say maybe what you're looking for in terms of performances, within a month you want these changes, or within two months you want these changes, but what's going to happen is that instead of saying, "I'm just going to kind of watch them and try to manage them over the next couple of months" well, in the next couple of months, you're going to be faced with the exact same decision, and you're going to rationalize it away again. Yeah, okay. In an ideal world, would you let them go today? Sure, but if you can let them go in a month or six weeks because you actually set these benchmarks, that's going to be faster than you would do it otherwise.

Jessi:

Yeah. No, that makes a lot of sense. Do you have a hard stop at 4:15 now? I just have one more question.

Annie:

No, I did, but my thing on the back end of this got moved to 5:00.

Jessi:

Okay, cool. I just have one more kind of example kind of question like this, and you kind of mentioned this, that these seem like small decisions, but they're actually kind of big decisions, especially if you're a six to seven person team. But sort of the bigger decisions, the quote unquote bet the company decisions where you're like, "Should we sell? Should we keep building, or should we go after this whole new market, and we're investing all our resources in that?" Is there a way that founders should be approaching those differently or different kinds of tools to be using for those things that feel more weighty?

Annie:

Yeah, so I think that it just has to do with how much time you put into the decision and how much of this pre-work you do. When it's a small feature that you're thinking about developing, whatever, just put an engineer on it in their free time and come up with some crappy thing that is going to get you the information you want, and I don't care if you go through a big process. I really don't, right? But if it's the direction of the company or should we sell, so the first thing is as much as possible try to think about these things in advance. On the should we sell question, you should be thinking kind of all along the way, "What are the circumstances under which we would sell?" And try to think about those in advance and get different inputs.

You have the founding team, maybe you have a CFO by this point. I don't know. You have investors who are very smart about these kinds of questions, you have board members and you can work with them to try to figure out how... Just send out to them, "Okay, imagine that we're fielding an offer and someone wants to buy us. What do you think the circumstances are under which we should do that? What does the offer need to look like? So and so forth." You can figure out what those things are, but as much as possible, try to do that when you're not fielding an offer, and it's the kind of thing that you can put on a cadence, you could do it every six months or once a year, you could say," Hey, let's go back and get that feedback on whether we should sell or not," because obviously that's going to be different when you're pre-seed than it is going to be when you're at Series E, what are the circumstances under which we should sell.

So as much as possible, try to think about those types of things in advance. When it comes to direction of company, again, think about what are the things that I need to understand in order to think about what direction I ought to take this company. Should I pivot? And that's going to be essentially an expected value question where you're going to be asking about how's it going on the path I'm on versus how might it go on the new path that I might take? And you should approach it as a new decision. So when you went into this, you were approaching it as a new decision and you should think about what are the costs to changing direction, what are the benefits of changing direction, what are the costs of sticking, what are the benefits of sticking? And you should try to, again, think about as I get a diverse set of points of view about this, what are the judgments that I'm trying to elicit in order to understand whether to do this or not, and get those judgements independently.

And then lay on top of that. When you're talking about switching, changing direction, you're going to be biased towards sticking and that's always going to be the case. There's a whole bunch of reasons why you're going to be biased towards sticking. So just understand... That's why I say approach it as a new decision as if you're examining those two options fresh. Imagine we were starting the company today.

Jessi:

Mm-hmm (affirmative). Yeah, I like that.

Annie:

How should we think about the directional choice of company? I hope that what you hear repeatedly through what I'm saying is that think about in advance what is the structure of this decision? Meaning what are the key variables that I need to be judging, what is the criteria that I need to be thinking about, what is the data that I need to go gather in order to think out this decision? And then make sure that you're getting those judgements Because no matter what, it's not going to be two plus two equals four. You're never going to get that kind of precision on directionally, which way should we take the company. I don't know. We've never done that before. It's going to be what people's best judgments of those things are, and it's going to be things about them judging which is going to end up with the most tailwinds in the market. Maybe you care about

Maybe you care about if we go this direction versus this direction, which one's going to have stronger product growth, or so and so forth. So it's trying to figure those things out, where are the unit economics going to be the best? I don't know. Figure out what those judgments are that you're trying to get from people, get it from a diverse group, not just the people on your founding team, but you can get it from board, you can get it from investors, which often have overlap, customers, so and so forth, and always think about that in advance and get the viewpoints independently. That's always going to help your decisions much, much more, and then again, just understand there are some decisions that you're supposed to take time with and some that you're supposed to go fast on, and if you can get that thing right, you're going to be so much more efficient in your decision-making.

You're going to spend your decision-making time on the things that really matter, and not on the things that don't. So one of the ways that I think about it is don't spend tons of time on your company logo, spend a lot of time on figuring out if this is the right product for you to be developing. Don't agonize over the name of your company, just figure out whether the product is going to work. But this is the type of thing that founders do all the time. What color should we paint the walls? Let's design our logo. What should we name it? Why are you spending so much time on those decisions? That's such a waste of time right now.

Jessi:

Yeah. Yeah, totally. One last thing, which I wanted to get your take on if you want to include this or not is I thought there could be one on when to quit, when to wind down your company. Because I know your new research or your new book is kind of centered around this theme a little bit, so we could kind of talk about that a little bit or put some of that in there if you want to, but totally fine not too.

Annie:

So that's actually kind of the same... It's the same issue. So it's the same issue as the employee question. So at the point that you're starting to think, "Maybe this isn't going very well," you probably should have shut it down a while ago. And you're probably not going to shut it down today. So I think that what's really important is first of all to say, "What are the things that I need to see happen over the next month, or six weeks?" And it's the same process, but now it's for you. What are the benchmarks that I need to be hitting over the next two months? Because you're always going to have this desire to say, "No, I know I can make this worse."

Jessi:

Or just I have to grind it out, I just have to work harder, kind of like Silicon Valley.

Annie:

Which you're going to like, "People won't respect me." But I'm just starting, so let me go through the three things. So the first thing is I know I can turn it around. Okay, so that's a big one. And the answer is maybe so, maybe not, but figure out over the next six weeks, what would be the signals that I turned it around. So one of the things that you can say to yourself, you can kind of do it both ways is imagine it's three months from now and I've realized for sure that I need to shut the company down. What were the signals that that was true? Or imagine it's three months from now and all of a sudden things have started to go really well. What are the signals that was true? And that can help you set some benchmarks.

So it might have to do with, for example, customer acquisition, how fast are you acquiring customers or whatever, but you figure out what the benchmarks are and just try to set those in a relatively short time window, and that's going to help you get over that because okay, so you might turn it around, but set benchmarks for yourself just as you would for an employee where you were thinking about this decision. So that's the first piece. The second piece is people will think less of me. They're going to think... The investors are expecting me to just grind it down to the nub and if I quit, people are going to think I'm a quitter and I didn't have the heart or the chutzpah or the nerve or the grit and nobody's ever going to invest in me again, and they're all going to think less of me.

So this has to do with a term that we'd use called external validity, which is what will other people think if, how are other people going to judge me if? So I have a fun story about this one. So there's this amazing guy named Ken Kamler, he was somebody, he started off as a microsurgeon for hands, and then he just sort of randomly through a patient got into climbing and he really loved it, and he was also a doctor, and then he kind of learned emergency medicine, and he started to go on different expeditions where he was a doctor on the expeditions. So he happened to be a doctor on one of the expeditions that was going up Everest in 1996, and when the disaster happened, that was Into Thin Air. He sort of ministered to a lot of those people.

But the year before, he has an amazing story about this issue of external validity. So they were climbing up Everest and the year had there been a lot of snow, so the going was very powdery. So think about it, the way that he described it is like they were climbing in a bowl of sugar. So you can't get any traction if you're climbing in a bowl of sugar. So you're going super, super slow as you're going up the mountain, and there's one part of the mountain that's very dangerous called the Southeast Ridge. I mean, obviously all of Everest is dangerous, and you have to descend the Southeast Ridge in daylight because if you don't, you're going to either fall thousands of feet into Nepal or thousands of feet into Tibet. Those are your options.

Okay, so there's nine people in the expedition and they're going to summit, and the going is really, really slow, and then at some point they get on their walkies because they have to have this discussion about as slow as we're going, if we get to the summit, are we going to be able to get back down to the Southeast Ridge in daylight? And so they're having the discussion and he describes it as remembering, he thinks, "Oh, my God. If I just turn around at this point, all my friends are going to think I'm a wuss and nobody's going to have any respect for me, and they're all going to I'm so stupid and I failed and I'm a failure and whatever." Anyway, the nine of them agree that they need to turn around because there's no way they're going to get back to this in daylight, and so they do turn around and they descend back, down the mountain. And as he says, when he got home, nobody said that. Everybody was like, "Whoa! That's amazing. I don't know if I could have turned around in that situation. You're a great decision-maker."

I think one of the things that we need to realize is that the things that we tell ourselves in our heads are generally to justify our own desire not to quit as opposed to what other people think. Investors aren't sad when you return capital to them, and if you do that, they'll probably invest in you in the future because it's a sign of great decision-making. I mean, you're sort of like, "I owe it to my investors to keep going," and it's like no, that's not true. You owe it to your investors to continue to invest money in something that has a high enough probability to win, that it's worth it for you to continue to spend their money. What you actually owe to your investors is to recognize when the project is done and to not continue on because you think they weren't respect you, because it's actually the opposite. They'll respect you more. This is something that actually Ron Conway points out, which I think is just a really good point.

And then the third piece of the puzzle that you'll hear is, "I owe it to my employees. They put their heart and soul into the matter," but that goes back to that thing of not wanting to close an account in the losses, and you're just using your employees as a way to say that. You've put your heart and soul into the matter. You're worried about having to close it at a loss like, "I lost. I didn't actually succeed at this thing. What does that mean? I'm a failure." No, it doesn't mean that, it means you were working on a startup where most of them fail. And from the employee's perspective, you're trapping them in a situation where they're working probably for equity and taking a hit on the actual cash comp that they're getting, and you've already in your head realized that the equity is probably worth zero. So why don't you free them up?

So this goes back to this thing of when you have the employee that you're thinking about hiring or firing, to reframe it as if I freed up the equity, if I freed up the comp that I'm paying them, the time that I'm putting into this employee to try to help them to actually get to where I need them to be. What could I use that for? Well, I could use that to hire someone who would be better in the job, and in this case, it's the same thing. I'm tying up this person in a job where they're working probably mainly for equity that I've now determined is zero. What if I freed up their time to go to another job?

Well, what could they do at that time? They could go to a where the equity is actually now worth something, it isn't worth zero, and that's actually better for the employees. So you think you're saying, "I don't want to let my employees down by shutting it down," but you're actually doing the opposite if you shut it down, you're letting them go to a company where they're going to find success, Where the company itself is going to be able to blossom and that's actually better for your employees, but we don't think about it that way.

Jessi:

Or yourself. You're free to work on an exciting new project or whatever.

Annie:

That's exactly right. You can go start something else, and because you returned the capital, your investors are going to invest in you again.

Jessi:

Versus pivoting to some random thing they would've never invested in.

Annie:

Just to not have to admit to defeat.

Jessi:

Yeah. Awesome. Okay, great. Well, I think that's most of what I need for now. I appreciate you spending all this time with me.