Episode 10

An inside look at the system that will outlast Bezos - Bill Carr & Colin Bryar on lessons from Amazon

Today's episode is with Bill Carr and Colin Bryar, former Amazon execs and authors of, "Working Backwards," a new book on the company's leadership principles and business processes. They tell the creation stories of the Kindle, AWS, and Prime in granular detail, and share insider anecdotes, from how leaders like Jeff Bezos and incoming CEO Andy Jassy operated, to Amazon's hiring process and key metrics.

Read the article that inspired this episode

Photo of Colin Bryar and Bill Carr
Subscribe with your favorite podcast app

Transcript

Bill Carr: [00:00:00] The challenge at any established company. When you go back to this question of how do other established companies build that other business to rival or exceed? The core business that they've got is that the demands of your core business will always soak up all of the bandwidth of your leaders. And even if one of them is assigned as a part-time job, in addition to managing some large business in the company, they'll also be responsible for some innovation.

It tends not to work to quote Dave limp, who is the senior vice president of the Amazon device business. The best way to ensure that you fail to invent something is by making it somebody's part-time job.

Brett Berson: [00:00:45] Welcome to in-depth a new show that surfaces tactical advice, founders and startup leaders need to grow their teams, their companies, and themselves. I'm Brett Berson, a partner at first round. And we're a venture capital firm that helps startups, like notion, roadblocks, Uber, and square tackle company building firsts through over 400 interviews on the review.

We've shared standout company, building advice. The kind that comes from those willing to skip the talking points and go deeper into not just what to do, but how to do it with our new podcast in depth, you can listen into these deeper conversations every single week. Learn more and subscribe today@firstround.com

today's episode of in-depth. I am really excited to be joined by two guests, bill Carr and Collin Brier. They are former Amazon execs who have written a new book that's out this week and on the heels of last week's big news that Jeff Bezos will be stepping down as CEO of Amazon. Their book working backwards could not be more well timed.

It's a great read for those who are curious about incoming CEO, Andy Jassy and the business practices and cultural foundation that will enable the company to continue succeeding without its founder at the helm. So just to ground our conversation, I wanted to share more details about why bill and Collin are the perfect people to explain the underpinnings of why Amazon works.

Bill started at Amazon in 1999 as a senior product manager for DVD video and went on to launch and run the prime video, Amazon studios and Amazon music businesses. Before he left the company in 2014, he then served as the COO of offer up before teaming up with Colin to write this book and start a consulting business to coach executives on how to implement the management practices developed at Amazon.

Colin joined Amazon in 1998 as the director for Amazon associates and Amazon web services programs, then he spent two years in an incredible role as Jeff Bezos's technical advisor or shadow working with the Amazon CEO on a daily basis to help him with business strategy, technology and operations in his last role at Amazon before he left in 2010, Colin was the COO for IMDV afterwards.

He spent a couple years in Singapore as the COO of an e-commerce company called RedMart, which was sold to Alibaba. What stands out for my conversation with the two of them is how Bezos and the leadership team invented not just slew of products, but an invention machine or a remarkably different way to run a company.

We start off by digging into why Amazon has uniquely been able to launch so many different seemingly unrelated businesses and how they've cultivated a culture of longterm thinking and true customer obsession, bill and Collin tell the creation stories of the Kindle, AWS and prime sharing granular details about the working backwards process, which centers around the now famous press release frequently asked questions document that helps clearly define the customer problem before anything gets built from an inside.

Look at how players like Jeff Bezos and incoming CEO. Andy Jassy operated up close to the projects that didn't work out and the inspiration Amazon's leadership team has taken from Toyota's manufacturing line. There's tons of fantastic details here to learn from. We get into what it means to be a great operator and the art of how leaders can learn to dive deep, but not micromanage, which given the name of this podcast was a topic we had to explore.

Finally, we end on what it's like to be a fly on the wall during Amazon's interview process and the mistakes bill learned from after nearly a thousand interviews as a bar raiser. I think bill and Colin do a particularly good job and not just sharing Amazon's practices, but highlighting what other companies can learn from them.

Whether it's their lessons on why innovation can't be a part-time job or the perils of taking a skills forward approach to exploring adjacent business opportunities or why mechanisms are more important than good intentions. There's a lot of food for thought in here for founders and startup leaders. I really hope you enjoy this episode.

And now Mike conversation with bill and Collin.

Well, thank you. You both for coming on the podcast. Yeah. 

Colin Bryar: [00:05:20] Thanks for having us. 

Brett Berson: [00:05:22] So I'm super excited about this conversation and the place that I thought we could start is at the 10,000 foot level and explore why Amazon has been uniquely great at launching a multitude of businesses that then ended up getting to meaningful scale.

And I think it's a great place to start because for so many large technology companies, this is an area that is exceptionally hard and I'd argue for most technology companies. They are thing. The thing that got them their first three, five or 10 years continues to be their one thing. And if they do launch an ancillary products or services, or what have you, they tend to.

Oftentimes not be material. And that one thing from day one continues to be that one thing. Right? So even if you look at a special company like Google, still there, one thing in terms of the core advertising that runs through search in YouTube, et cetera, it tends to be their one thing, but Amazon has done a uniquely great job of launching this horizontal set of products and services and businesses, and would love to talk about how that works.

What are the underpinnings that enables Amazon to launch such a diverse set of businesses that in many ways seem unrelated, require in some ways different cultures, right? Pick, pack, and ship is different than building infrastructure for technology companies to build on top of, for example, what's special about what Amazon does.

And what can others potentially learn from that? Colin, why don't you 

Colin Bryar: [00:06:53] start as Amazon is in they're very different and have different characteristics, skillsets, AWS, primarily B to B, you have the e-commerce business, you've got pure digital delivery, huge logistics, business, and so on. But what a lot of people don't really focus on is what do they have in common?

And there's one thing in particular that all of those businesses have in common is they were built the same and operated using the same process. And in our book, we do call it the invention machine, but really at its heart, it's Amazon's 14 leadership principles surrounded by five very specific processes.

And those five processes are how Amazon hires, how they organize into single-threaded small separable teams, how they communicate using narratives, the working backwards product development process, how you take new ideas and make them a reality. And then finally, how does Amazon continually measure and operate the business by focusing on input metrics?

So all of the businesses, prime, AWS, they started from that invention machine, really from ideas on a whiteboard. And they've grown very large, but they've stuck to those set of principles and processes. And other part about what makes Amazon different and unique is that if you had to describe what is the.

Culture. It's really, long-term thinking longer than most companies, a longer time horizon than most companies. It's customer obsession, not just customer focused, you know, that's woven into everything. Amazon does the spirit of invention, and that also means that you need to be able to handle failure and how you handle failure in the company is very important too.

And then there's pride in operational excellence. Most of the stuff that people do on build that's under the tip of the iceberg that customers never get to see, but that's where a lot of the magic happens when, uh, you know, you see these crisp things. How does this package come in? 12 hours after I click reliably day after day.

And the long-term thinking especially is different than we've seen in most companies, along with customer obsession, because a lot of people think in order to move fast, you need to always iterate very, very quickly. And not stop to think about what is the value that I'm creating for the customer. What is the problem I'm trying to solve?

And especially when you move into a brand new area, those questions are hard to answer. And it usually takes several iterations by focusing on the customer experience, rather than just throwing stuff over the fence, launching it, or throwing it in an app and seeing how it sticks. And so Amazon goes through, they have a deliberate process to go through and all of these businesses went through there to make sure that it's a well-defined customer problem they're trying to solve.

And then the solutions everyone is on the same page about what they're trying to do. AWS went through this in 2004, 2005 timeframe where. We had an inkling that more than an inkling, that web services really were going to, it was a new paradigm on how software engineers would build and deploy software.

We didn't quite know how to expose it to the outside world to developers. We were working internally across several different groups, experimenting on what worked well for using web services internally at Amazon to deploy and build software. And, uh, so Andy Jassy started a small team to go start investigating that.

And for 18 months it was pretty much writing. Documents doing some prototypes. And we're meeting with Jeff Bezos to say, what are the problems that we're going to try to solve? 

Brett Berson: [00:10:36] How do you even figure out that that is an area worth spending 18 months exploring, learning about writing about and refining, right?

There's a near limitless number of things. And it also feels so, so adjacent. If you think about something like prime and prime video and the stuff that sits around the core Amazon consumer experience, I could see waking up every day and saying, okay, what do our customers care about? Okay, this is adjacent.

This is in Jason. This can be self-reinforcing. Whereas a project like this seems so far a field. 

Colin Bryar: [00:11:08] So a couple of things. One is we were getting some validation from a couple of different areas from developers. That web services were just a better way to build into play services. We had exposed Amazon's catalog for instance, through a set of XML services and what we found relatively quickly, as some of our internal developers were using this external.

Service, which is meant for third-party developers to build the Amazon website. And they just did that because it was better than the internal tools that we had built. And then we had a lot of small separable teams at that point coming to Rick Dalzell, the CIO Jeff Bezos saying can build our service, but it takes too long to integrate it into this infrastructure and then go deploy it.

We're spending more time doing the same thing over and over again. We were also working with a set of partners, building websites for third parties at that time. And we found that web services was the best way to, and the fastest way actually to go build another stack other than amazon.com, where there were some data points.

And this is where that spirit of invention comes in. We weren't certain with a lot of things like how big this could be. Who else may be doing that because if you wind the clock back to that time, there probably a list of other companies who are in their own right, should have been racing fast to get there and have had the capability and the customer knowledge.

We were not a B2B company at that time. And our customers at Amazon were either sellers, third party sellers, or, uh, consumers. And so, um, it was a relatively small team to start off. And this is just where the spirit of invention comes in. Hey, it's worth trying, it's worth pursuing and let let's dig deeper.

And it was an iterative approach. And as we settled in on the two, first things we wanted to roll out were compute and storage. We refined that over the period of 18 months and then launched it and constantly got feedback from customers. 

Brett Berson: [00:13:07] When the team is thinking about what projects to kick off or areas of exploration is.

A part of it. What is Amazon's unique ability to do something in this space or does that really not matter? And it's where is there an opening where's there an unmet need and our existing footprint capabilities, whatever. We don't really care. Maybe bill, you could jump in here. 

Bill Carr: [00:13:32] Yeah. That's actually one of the other remarkable things that other companies can learn.

So most companies take what you're describing, which is called a skills forward approach saying, Hey, we're good at e-commerce or we're good at retail or whatever it is that company is good at. So they look for new businesses, adjacent businesses to build a leverage, the strengths that they already have when I was in business school.

This is frankly, one of the things that I was taught, but successful companies do. And it's one of the many ways in which Amazon succeeded by taking a peculiar or contrarion. Approach to how it thought about managing and building new businesses. So AWS and digital media are two good examples. So I was one of the founding leaders of the digital media group and the reason.

We started to work on digital media was because this was 2004, beginning of 2004, and we just finished calendar year, 2003. And at that time, Amazon had about 5.7 billion in revenue, which was certainly a big number, but the bad news was that 77% of all of Amazon's global revenue back in 2003 were for physical media products.

You know, hardcover books, paperback books, CDs, DVDs, VHS tapes, and the writing was on the wall. By this time that the world was going to move to digital media. There were already more than a million iPods sold Napster had millions of people, sharing songs. People were ripping their CDs and creating playlists.

It was clear that the days for physical media were numbered. So what Jeff did is my boss, Steve castle, and I, he literally pulled us off of what was then the largest business in the company, the physical media business, and said, you two now need to go figure out. How we going to build a digital media business.

And the first notable thing is that neither Steve and I knew anything about digital media and ironically, Steve was in some ways, so non-techie that he didn't even have a stereo in his car. And so we seemed like two unlikely people to go figure this out. The next notable thing is that like the AWS process, we really literally spent more than 18 months then coming up with different product ideas, refining those ideas, and eventually using a new process that we developed through that time called the working backwards.

PR FAQ process. We would write press releases that described new products. We wouldn't worry about whether we had any idea how to build that product at the time, but we just said, does this product sound like something that people would really want? And we wrote many, many of these tens more, probably more than 50 different ideas and eventually focused on the best ones.

The standard approach would have been for us to then go build our own knockoff, iPod and iTunes competitor. But we intentionally did not focus on that because we wanted to. Invent something new on behalf of customers to create real value. So instead we were drawn to this other idea of how could we create eBooks was, uh, was sort of a nothing business at that time.

You could only read an ebook on your PC or Mac and the prices were high and oh, by the way, they were a pitiful selection of eBooks, maybe 15 or 20,000 eBooks available as opposed to the hundreds of thousands of books in print. So there was no real good reason for people to actually buy and use eBooks back in 2004.

But we try to envision what, what would change that? And through an iterative process, we came up with the idea for the Kindle and when we came up with that idea, several things were true. Number one, we had never built a hardware device as a company. We sold hardware devices. We sold TVs. We sold a lot of electronics as an e-commerce retailer.

But we certainly had no capability to build one. And I remember questioning our wisdom at one point, when we decided to go do this and say like, why on earth would we go build our own device? We don't know anything about that, but instead of taking a skills forward approach and say, we're only going to do what we could do, we took that.

Customer obsessed approach to say what would be the product that if we could create it, customers would love it. And the other thing that would have been easy to do is to outsource it to an OEM or ODM, to design and build this for us. And instead we said, no, no, no, no. We need to actually hire up a team of people who are experienced hardware, engineers, software engineers on devices who have designed built many successful devices over time and hired a leader named Greg Zare from Palm to do that and build up that team.

And this was also unconventional many senior leaders in the company questioned the wisdom of such a move. And this approach by the end, it took a lot longer Kendall didn't launch until 2007. So more than three years after we got started working on digital media. But by really thinking through the detailed customer experience, by making sure that those details would really delight customers, some things simple that were hard to do, like when you opened up your Kindle, it was delivered to you.

It already knew. It already had been associated with your account. So any eBooks that you had bought in the past were already downloaded to your device or the fact that the device was always connected to the internet. And there was one way to shop, download, and start reading on one device. These were all things that were breakthroughs at that time.

And we only accomplish them because we took that customer obsessed approach and then sought out, develop and built the skills we needed to get there. When 

Brett Berson: [00:18:58] you think about that starting point. So bayzos says we need to work in this area, potentially drop what you're doing and go explore how much when you think in reverse.

So you think of all the seven, 10 major successes started with the senior leader saying, Hey, somebody go explore versus something even more bottoms up or organic. 

Bill Carr: [00:19:23] So as we already described two or three, depending, if you want to count digital mean devices, one or two businesses, and as well as AWS, they were started by, yes.

Let's create this group to go. Explore develop and figure out what to go build there. Prime was a more organic process through trial and error. And so one of the other hallmarks of Amazon is the fact that it has this attitude of being stubborn on the vision, but flexible on the details and the way you can see that as through how did prime come to be?

So prime was developed to solve a very specific customer problem. And the customer problem was the customers don't like paying for shipping. And it was not hard to figure that out. All you had to do was field, any customer survey, all the way back to the beginning of Amazon to ask them, what do you like least about shopping with Amazon or what would make you want to buy more?

And number one or number two would always be, I don't like paying for shipping. The hard part was figuring out well, how, since shipping costs real money, how do we actually afford. To do that. How do we construct a program that would enable that? And there were various efforts, various free shipping promotions, and then one that stuck the longest was one called super saver shipping that some of you may be old enough to remember where for many years, the way that you got free shipping on Amazon was to place an order for $25 or more.

And then in that case, part of your trade-off was that the items would be delivered to you a little more slowly than the standard speed. So it would take two or three days longer than if you were to standard shipping. But fundamentally the company in 2003 was hitting a wall on its growth. We recognized that media business a its future was not looking bright, but more importantly, in fact, the year of your growth was decelerating.

And a lot of it was because while a lot of people use this super saver shipping. Yeah, you had to get over this hurdle. So lower ESPN is under $25, you know, just want to one of those things, it wouldn't qualify. And two people don't like having to wait for stuff. So we try to figure out, well, how can we solve this?

How do we get to where people can get stuff fast and free? Colin can jump here and tell more about sort of that iterative process. It was a process of working through a lot of different business models, a lot of different ideas before landing on one that became prime that in this case, actually, ultimately there were a number of people that contributed, but Jeff was really the person that finally came up with that idea and pushed for us to go do it.

Yeah. 

Colin Bryar: [00:21:46] I'll just add that most of these ideas, it was not, Jeff said, let's go create cloud computed, or we're going to go create a Kindle. Everyone knew that the physical business media business was moving over to digital. The trick is when do you do it? And how do you do it? And those things for collaborative efforts, same thing with the cloud computing, there were multiple data points.

And with prime also, there was some, we knew. Yeah, customers want cheaper products. They want them faster and they want more selection. There wasn't any surprise there or deep insight that only one person at the company had. The trick is figuring out how to do this, how to organize. And as bill said, how to afford it, 

Bill Carr: [00:22:31] right?

And then the other example is fulfillment by Amazon, you know, which is a huge business for the company, enabling merchants to outsource, warehousing and fulfillment of their items that they sell on Amazon and other retail sites. And that idea, again, it wasn't the Jeffer, any one person. It was through talking to our merchants and observing what were their challenges and thinking about our capabilities.

I think it was apparent to a number of people, what this could be. And for a number of years, this project bounced around the company has something called self-service or fulfillment and. What's also notable to learn is this is true at a lot of companies where people can see, oh, I think there's this opportunity and there's this project and let's go do this, but the project wasn't happening, it just wouldn't launch it.

Wasn't launching an Amazon. It kept being on long lists of initiatives to do it. Wasn't getting worked on it. Wasn't being resourced appropriately. And it wasn't until Jeff Wilkie. Now the outgoing CEO of Amazon's consumer business assigned one of his strongest VP lieutenants. To go stop, drop what was a very large role in the operations organization and go focus fully and completely on what was then known as SSOs self-service order fulfillment and you know, in doing so then this VP leader was able to actually identify, well, what resources do we really need?

How do I make this project actually go and then hired and developed the right team and finalize the products back within a matter of months actually was able to launch that program, which we now know is fulfillment by Amazon. So the other important lesson is that a lot of good ideas can get trapped inside big companies, even medium-sized companies, because they're not staffed to succeed.

And Amazon came up with a unique solution that it calls single-threaded by organizing by what's called single-threaded leadership for. Each important initiative, having one leader who works on that initiative and nothing else as well as their team and in doing so that gives them the appropriate focus to make sure that they can devote time and attention to make that a success.

Because the challenge at any established company, when you go back to this question of how do other established companies build that other business to rival or exceed the core business that they've got is that the demands of your core business will always soak up all of the bandwidth of your leaders.

And even if one of them is assigned as a part-time job, in addition to managing some large business in the company, They'll also be responsible for some innovation. It tends not to work to quote Dave limp, who is the senior vice president of the Amazon device business. The best way to ensure that you failed to invent something is by making it somebody's part-time job.

When you 

Brett Berson: [00:25:21] explain the process to ultimately launch Kindle on them, what would become the digital media business? You talked about this 18 month voyage of discovery that was anchored in the, the concept that I think was pioneered at the time, working backwards, press release FAQ. I think it'd be interesting to talk about that in more granular detail.

Like if you were to look at any given week long before you landed on the Kindle, what was going on? Maybe some more detail about what that documentation looked like. And then if you had 50 of these press releases, what is the process that you called and prioritize and decided, okay, Kendall's it. And we're going to make a huge bet here.

Bill Carr: [00:26:06] It's a funny story. That's really 2004. I'm starting down the path of being one of the leaders here of this new digital business. And so Steve and I go off and we start to develop our plans for what are we going to go do? And what are we going to go do? Not only in eBooks, but digital music and digital video, which didn't exist at all as a business, then we both had MBAs.

And so we pulled out our MBA toolkit and we did what you're supposed to do, which is you go look at industry analysis that say, what are the projected sizes of each one of these market segments in the future years? How big will the ebook market segment be? And you come up with some projection for what our market share will be.

You have each one of them by year, you build a pro forma PNL factoring in like what's going to be our cost of goods sold. What would be our other sort of, you know, Fixed and variable costs. And we took that into Jeff and said, okay, great. So here's kind of what the business will look like. And so we're ready to, you know, go ahead and get started and hire out the team and go start negotiating rights agreements with our partners at the publishing houses and motion, picture studio, some record companies.

And Jeff just looked up from these documents and spreadsheets and looked at us and said, where are the mock-ups. And, you know, by that he meant, you know, where are you your mock-ups of the website experience or how the customers will actually buy, interact with use, listen, to read these digital media products.

And we had not done any of that work at all yet. It's actually a lot of work to figure that out. So we kind of went back to the drawing board and tried to figure some of that out and develop some mock-ups and we came back and that didn't go very well either because frankly, we hadn't really thought through all the details.

And by the way, we were really thinking very big. About what we were going to do. We were thinking about a kind of just an extension of our physical retail business is just sort of more items to add to our catalog if you will shop for, and we hadn't really factored in the way in which digital media was so different and that our role as an aggregator in digital media, wouldn't really confer much value to consumers.

That was great for physical goods, by being able to aggregate, you know, millions of items in one store and ship them to your home, like that was hugely valuable, but it had no real value in digital media. And we hadn't really unlocked any of the problems to make that digital media useful and valuable because in those days there's no good bandwidth streaming didn't exist.

Didn't have devices, all, all these other problems. And so Jeff basically by asking us about the mock-ups and challenging, our assumptions forced us to push ourselves and our thinking out to the ends of the value chain and think much bigger. About how to do that. And so it was a halting process though, to figure out like, how should we as a work process, like how do we have productive meetings and ideation to go do that?

And so mock-ups, didn't really work because it's a lot, it takes a lot of time and effort. To work with a designer to come up with a good mock-up and you spend much more time building the mock up than actually thinking through all the issues and details of the customer experience. So in the end, what we figured out was a much more effective mode was frankly, just to write down in a word document, describe the customer experience you were going to go build and in that document to clearly articulate and explain, okay, what's the problem.

I'm trying to actually solve for the customer, because I know by the way, for each one of these businesses, there are many different problems you could choose to focus on which problem or problems are you going to address. And then what is the product you're going to build to solve it? And it wasn't until Jeff said, everyone, just go write a document, you describe your product.

And then we came back to the room and probably spent four hours going through, I dunno, 12 different product ideas, a huge range of ideas, but now we were actually getting somewhere. Now we could actually compare and contrast different ideas. And it was after doing a few of these meetings at some point, Jeff landed on the point of like, I know to really make this process better.

Let's just write the press release. And he said, the reason to do that is that normally happens at the end of the process. And typically in fact, the product engineering teams long ago, years, earlier months earlier made decisions that will affect how the product is sold and marketed. And then just. Just tossed over the wall to marketing, to then figure out how to sell it.

And he said, let's instead start at that end because of the press release doesn't if in one page press release, you can't describe something that sounds super compelling that people really want and need. Then there's no point in building it and it doesn't take that much effort or time, like unlike building prototypes or unlike building mock-ups.

Or in like launching an MVP, anyone can write a press release and you can write many of them very, very quickly. And then you can form a meeting to review lots and lots of them. And if you spend the time, then when you find a good one, you can refine that idea. So that ability to, uh, that doesn't require them that many resources to write one, and you can actually compare them that allows you to over time, look and see which ideas are the best.

And they float to the top. Once you've actually landed on the good press release, then you do the second part, which is called the frequently asked questions. And the short answer on that is those are really designed to help then address. Okay. This thing that I just described as a product doesn't exist today and there undoubtedly will be some challenges and problems to build it.

Well, what are they and what solutions have I come up with or do I need to come up with to actually build 

Brett Berson: [00:31:27] it on the point? And you follow your curiosity, think about the customer and you start writing these press releases, and let's say you have 10 or 20 or 30 and you have a meeting. Is it just judgment and gut, oh, this seems really interesting.

Or that, that you end up then starting to drill down in one 

Colin Bryar: [00:31:42] area. Yeah. It's pretty apparent when you read the press releases, which ideas have staying power and which ones don't often you combine a couple of ideas from different alternatives. It's hard to write those, but once they're written and if they're written well, it's easier to judge that then looking at a bunch of spreadsheets and financial projections and cost projections.

That really quite frankly, the error bars are so high that you could make those say anything that you want well-intentioned, but there's so much risk and uncertainty that those don't actually help you make the right decision. What helps you make the right decision to take the next step? Is, is this the customer problem we want to solve?

And is this the customer experience that solves that problem? 

Brett Berson: [00:32:30] And so if I were to look at what ultimately became the press release for AWS or starting with compute, what did that read? Like, do you remember. Well, 

Colin Bryar: [00:32:40] we can talk about compute. Is it at first, started off as provisioning because at this time Amazon was operating a data centers and if a team wanted to build the service and deploy it, they actually had to spec out hardware.

They had to justify the cost, how many machines they need. They had then, you know, we, whether it was all in stocker or, you know, the type of machine, the, the CPU and memory requirements, and then deploy that to one or more data centers that was huge, heavy lifting that we had to do over and over again. So the first iteration and was started by Chris Pinkham and Ben black were two of the primary proponents of this idea said, let's try to make provisioning a service instead of a multi week exercise for every team.

Uh, so the first. Press releases. They weren't press releases. There were more, but they were documents that said here's what this provisioning services is going to be. And it migrated to, well, we're talking a little bit more about just building and deploying hardware. It's more execution services. And so morphed from that point.

And then eventually it became compute and that's really was the, and even to figure out, well, what is the fundamental dimension or unit of measure that we're working with? It took a while to get there. It was a path and, and really what the path was. It was uncovering the truth about what are we fundamentally trying to do here and to solve and to make customers' lives better.

You don't often know that, especially in a brand new area when you're starting out and you have to be okay with that ambiguity. And this is where the spirit of invention comes in to say, well, we're going to go explore it. Maybe we'll find something. Maybe we won't, or maybe we'll find something. And it turns out it's not big enough.

So it's not actually worth 

Brett Berson: [00:34:28] doing sort of that case it's not worth doing right now. Is there the top three reasons why as a team you end up there or it's, there's a long tail of a zillion reasons why something gets. Shelved, is it 

Colin Bryar: [00:34:40] big enough to move the needle and not move the needle? Next quarter, is this big enough to move the needle?

You're a feeler use. The long-term thinking lens. That is one reason another one, you know, sometimes the technology just may not be ready yet. So before those Kindle meetings, before Steve and Phil even formed the digital group, if you wind back six to 12 months before when I was Jeff's technical advisor, We were looking around debt display technologies, and they just weren't available.

We had actually talked with the ink folks, right, as it was still a research project out of the MIT labs. And it wasn't in high production volumes. You know, we thought there was something there, but it just wasn't ready. The bad ideas are pretty easy to filter out. It's just, you have a limited set of resources.

And what, and where do you want to place your bets with your bottleneck resources to really move the needle and make a big impact either in an existing business or a brand new business altogether? The 

Bill Carr: [00:35:44] other class of PR FAQ's that wouldn't move forward would be ones where you read the PR. And the product sounds great.

The solution sounds big. It sounds like it would be a very valuable thing, but you may have identified though one or more challenges or constraints and how to actually pull it off. Like, for example, if you want to enable a way for people to shop across every little local retailer. In the United States of the world online, you know, that's a pretty cool idea.

And oh, by the way, a lot of companies were working on that for a long time. But the reason it doesn't exist is because there's a difficult constraint there, which is how would you actually source real-time accurate inventory information from lots of little small and medium sized businesses and solving that problem.

There's no a known scalable solution to that problem today, but if you could solve it, you'd open up and unlock a lot of value. So one of the great uses of the PR FAQ is actually identifying a ha I have found something, a problem, and a product solution. That would be awesome. If I can just focus on now, here's the challenge to unlock it and then focus the team on, okay, come up with ideas.

For how to solve that challenge. That's one of the ways the companies unlock a lot of value is Jeff would often say we should never shy away from solving hard problems because some hard problems that if you solve them can unlock a lot of value. Yeah. 

Brett Berson: [00:37:17] In these journeys of exploration, one of the core outputs is this series of PRS.

What else is going on over that 18 months? Or if you think about the AWS journey or digital media. Other than somebody having an idea and clarifying it and writing it in that format. What else is going on in this exploration phase? Zoom in on one week, what is actually going on? 

Colin Bryar: [00:37:40] I can speak for AWS. There were some things that the team had to invent in specifically in terms of how do you have a network effect when it scales.

So as S3 gets bigger and bigger, how does it get cheaper, more reliable, and better as it grows versus. Oh, my goodness. We've got to just keep adding hardware and, you know, our costs they're going up or they aren't, they aren't declining. So there was a lot of research that the current CTO Verner Vogels had brought into Amazon and the technology was finally ready to match was in academia with the capabilities and that took a while to solve.

And so some of the things about, yes, we want to build this simple storage service. A lot of those meetings were well, how is it going to scale? How is it going to fail and what types of failures are acceptable and what types aren't. And can you predict the reliability in advance? Those are very, very hard engineering problems and you don't just solve those by writing a couple extra paragraphs.

You have to go and do some research in between. And for the Tyndall I was kind of on the other side of the table is bill. When I would come to these meetings with Jeff as his technical advisor, just looking at different. Prototypes or can you actually put a radio inside the Kindle? How much is it going to cost to have this always on?

Is that a feasible thing? And if we do it, here's what it's gonna look like. Holding styrofoam, prototypes of different form factors. Once you get the hard questions, you have to go do the research experimentation or development to figure out is this a feasible approach that we're doing? Does that 

Brett Berson: [00:39:22] get shaken out in the FAQ process?

You know, someone will ask a question and then somebody say, oh, that's a good question. And then somebody will go off and research it and then come back. Or is it more organic in the, yeah, there's five things we've got to go look into and Jane, you're going to do this, Bob. You're going to do that. You'll 

Colin Bryar: [00:39:36] find out that sometimes there's just a tough.

Question that no one had thought of. And to go back to Bill's example about, well, how are we going to integrate with the point of sale systems for, you know, hundreds of thousands of retailers. We need to come back in with some proposals on how to do that, how expensive it is, but this is where the iterative approach actually works quite well is because you're, even if those they're hard questions to answer, figure out it's still a lot cheaper to answer them at this point, then later on in the project or when you've actually started building some things in some of the paths that you may have otherwise want to do, Explorer are blocked off because of either design considerations you made or choice of what you want to build and partner.

Bill Carr: [00:40:21] Yeah. It exposes the key assumptions and risks. People have a tendency to want to sort of start writing code, start heading down the path and launching quickly. They will not necessarily look. Carefully at the various hurdles that they're likely to encounter along the way. And anyone who's listened to a lot of people describe a different business plan.

No. The difference between when a team has a good answer and a solid plan for how they're going to overcome a hurdle versus a team that is either dismissive of and, or doesn't have a well thought out plan to overcome a hurdle and no differently than the way experienced VCs, look at the various proposals that they're screening.

And, you know, they know the difference between those two things. It's no different with the PRF IQ process. Do you want to fund. The proposals that are really well thought out and where they've already got credible plans to overcome the hurdle, just like any venture capital investment for some new product, you don't never know really whether it's going to work or not, but a, you have a common understanding among the team and the senior management of exactly what is the problem?

How are they going to solve it? What are the real challenges? And the beauty of dealing with all these things upfront is then it allows the senior leadership team to then feel clearly aligned with the team that's going to doing the work rather than having to worry about what directions they'll wander off in after they give them the go ahead.

These 

Colin Bryar: [00:41:54] businesses and decisions are complicated. And a lot of the things that Amazon has put in place are really just stuck the odds in your favor, but at the end of the day, if you want to go truly run an experiment, it's not an experiment. If you know, it's going to work beforehand. And so you have to be prepared that, okay, we've done all we can burden and go try this and it may or may not succeed.

And then you make the decision. Do we want to go for iteration number two, or do we want to stop and move on to something else? So fire fountain is a good example. There was no fire phone, too. There were other parts of that project that survived, but I can tell you with the digital examples, if the first iterations didn't take off, we would have stuck with it.

Because as bill mentioned earlier, 77% of our business was a good chunk of that was going to go away. It was going to be cannibalized and it better be Amazon that was going to cannibalize that business than a third 

Bill Carr: [00:42:50] party. So the other really important thing about this by having this well-documented PR and FAQ is that by having that and iterating on it with the senior leadership team, when they say go, everyone is operating from the same basis of understanding the risk and they're agreeing to it together.

So when, or if that project fails, they're not going to blame the team for like, oh, that was a dumb idea. And you're fired. They all agreed to go do this idea. There may be issues of execution that might be a performance issue. But one of the healthiest things that the company does is that most of the new leaders are there because the, frankly they've actually had several failures and the right kind of failure where you learned something and then took away from that and don't make the same mistakes again is celebrated in the company.

Whereas in a lot of companies, the team or the people that fail, like then they're looking for jobs. I want to 

Brett Berson: [00:43:47] come back to some of the five processes that Colin was talking about as we kicked off, because we spent a bunch of time talking about the PR FAQ process. Before we do that, I was interested in having Colin, maybe share a little bit about the experience in being Jeff's technical advisor, and maybe some of the surprising things that you left that experience learning or frameworks or approaches or ways in which you think about products or companies or businesses or those types of 

Colin Bryar: [00:44:15] things.

First of all, I was incredibly fortunate and lucky to been in that role for two years, working side by side, with Jeff and also the Amazon leadership team, the esteem it's called basically Jeff's direct reports. I mean, a couple of things I took away from that job and still have seared into me is that long-term focus customer obsession, invention, and also how to be a great operator.

Some surprises, there are some things that you can learn and teach, and then there's some things that you can't, you know, you can't teach someone to be. Taller or even smarter, or, and you can also, at some level you can't say have better intentions going into it. Cause most people actually are trying pretty darn hard and they have good intentions.

So what Jeff was always good about is saying, okay, here's an issue or problem that we encountered. Do we have a mechanism in place? So it doesn't happen again because of this high-performing well-intentioned person or a team tripped up or made a mistake. There's probably a process that needs to be fixed and these can be lightweight things to it.

It doesn't have to be huge heavyweight big company processes in place to learn verbal and teachable skills are how to hire well and how to be a better operator. 

Brett Berson: [00:45:37] So on the topic of how to be a great operator, I guess the logical question is, can you tell me how to be a great operator? Sort of explain that to me.

Colin Bryar: [00:45:46] business, or even if it's a small group or full company, you could, you just think of it as a process. It can be a complicated process, but inside there's this black box and you have some of these outputs of whatever you're trying to do. So this machine or process spits up outputs like revenue profits, and to be a good operator.

You can't just focus on those output metrics, you know, in terms of number of customers or growth rates, you actually have to instrument and understand what this process or business is. And by doing that, you need to identify what are the controllable input metrics, which means what are the things that I have control over that?

If I do these things right, it's going to yield the desired result and my output metrics. And I think that the best operators I've seen one is. They knew very clearly that if you push these buttons or you turn these levers in the right way, that you're going to get the results you want, they understand that process through and through, by the way, those, those input metrics are usually customer related metrics.

What is the customer experience? Is it getting better this week than it was last week? And you put in an inordinate amount of focus on the input metrics. A lot of people say Amazon doesn't really care about profit or growth. I think that the data say otherwise, but we're the main focus is, is on those input metrics and it's harder than it sounds, you know, to figure out well, what if I do these.

10 or 20 things, right. Will things work out? So you have to experiment a little bit and make sure that you have that as a leader of a group or an organization. You need to clearly explain what those are and then measure it day in and day out. If you don't measure something it's going to go wrong. So, you know, and the last thing a great operator does is they instrument.

They put in little measurements all along the way to know exactly what's happening. Yeah. 

Bill Carr: [00:47:46] Expand on that a little bit more. One of the Amazon leadership principles is one called dive deep. And when Robin Andrew Lovett, she was the HR leader who helped actually go out and interview and understand the different leaders at the company.

And what were examples of the embodiment of great leadership? One of her role models was Jeff Wilkie. And at that time he was the senior vice president of worldwide operations of the company. He's now the outgoing CEO of the Amazon consumer business. And Jeff had a career working at companies like allied signal in operations and process.

And so she wrote that that leadership principles basically, you know, observing him and how he operated, which, and that leadership principle says that leaders operate at all levels and they stay connected to the details that audit frequently are skeptical and metrics and anecdotes differ, and no task has beneath them.

So that leadership principles, one of the ones that defines great operator. And then I'd say the other thing just to add, which column touched on is one of the senior leaders at Amazon met with once Diego, Piacentini the senior vice president of, of international. I remember remoting some particular problem or issue with my business one-on-one with him.

And he was actually talking to me about taking on some new challenge and I was. Trying to describe why I didn't really think I was, you know, all the ways in which maybe I'm not up for that challenge. I said, bill, look, everything can be distilled to a process. And what he observed in me is like, you've now shown a track record of creating and running things based on process.

And so long as you apply that here too, you'll succeed. And I, that stuck with me and I realized as I left Amazon and started working with other small companies, but the big difference between sort of the challenge for a founder is that making that leap to understand that everything in the company that's important can be and should be distilled to a scalable, repeatable process.

And that observing great leaders at Amazon like Jeff Wilkie, like Diego Piacentini, who spent their time doing that. I learned how to be a great operator, too. 

Brett Berson: [00:49:52] Point of diving deep as leader. Is there a line or a way in which that's done that doesn't border on micromanagement or is micromanagement kind of a misunderstood thing that actually can be hugely beneficial?

Colin Bryar: [00:50:06] I'll give you an example as was in 2010 with the annual planning process. So the S team sets, so they call them S team goals. There were 452 of them, and they're very, very detailed goals. They were largely input driven, but they were on the order of, in this geography. Here's how much selection we're going to add for this category.

Or we are going to bring down the latency of this AWS service from X to Y. It is that level of detail that Jeff and the S team has direct got to. And so some organizations as they grow the leaders think, well, my job is just to focus on big picture at Amazon. The leaders are expected to know at that level of detail, their own business about what's happening, because if you forget those and you don't know is my business better this week than it was last week, this quarter than it was last quarter.

It's probably going to go. We're set some time on undetected and that's what diving deep is about. It's not really micromanaging. It's staying on top of the details of your business. Looping 

Brett Berson: [00:51:16] back to the input versus output metrics concept. Can you give one or two other examples of what that actually looks like in practice?

And also seems like depending on how you frame it, a given input metric could be an output metric for something else. 

Colin Bryar: [00:51:31] There aren't formal definitions, first of all, but some of the basic input metrics to Amazon's retail business are what are the prices down to the product level compared to what's out there on the market price competition, how many new items were added and in stock and ready to ship via Amazon prime every day.

And you weight that based on traffic instead of just overall items, what is the average delivery time and what were the number of orders where Amazon missed promises? What are the number of defects? A great input metric that doesn't really see the light of day that Amazon pays a huge amount of attention to as inventory record accuracy.

And what that just means is if I have an item, is it actually on the ILO bin and shelf that I say it's on, that's hugely important. Especially if you're say I want to ship this a customer orders, an item, and you need to know where it is and you're going to ship it out in 30 minutes. If you know, someone goes to pick that order or a robot goes to pick that order, and it's not there, you've missed that customer promise.

So those are all types of input metrics where output metrics, if you will, what was the, what revenue did I do yesterday? What was the gross margin for the electronics category for the month? Yeah. And 

Bill Carr: [00:52:51] what separates a average from a great operator are ones who really dive into these in great detail and.

Tear them apart and are skeptical about whether that metric is really measuring what you mean it to measure are constantly deriving or devising new metrics based on their observations or based on when problems or defects occur. This is actually one of the best times to really understand, okay, well, why did I have this big customer problem and why didn't I have the right metrics upstream to detect that this was going to happen?

And so now what new metrics do I need to develop to be able to monitor measure and prevent this from happening again? So as collum pointed out, it's not by luck that Amazon delights so many customers, there is enormous inordinate amount of operational rigor that is happening behind the scenes. 

Brett Berson: [00:53:52] On that related note, the topic of customer obsession.

Comes up time and time again. When people talk about Amazon, you both mentioned it when you read shareholder letters. I think a lot of companies talk about the customer's always right, yada yada, yada, but there's obviously when you talk about it and the way that it instantiates itself at Amazon is obviously seems different.

What is special about it? And maybe if you would compare and contrast with other great companies, what's different about the way that the company lives, the concept of customer obsession. 

Colin Bryar: [00:54:29] Some of the fundamental processes are designed with customer obsession. First and foremost, you know, we talked about the working backwards process.

The very first thing you focus on is what is the problem I'm trying to solve for the customer and how am I going to solve it? Any new idea that you're developing the customer is front and center in everyone's mind, and you don't move on until you get that right. That you know, these metrics, the bulk of them are describe how did the customer use my product or service?

I like to use the analogy when you're a sole proprietor in a coffee shop, you get to see how long the lines are. You get to see if people think the coffee's too hot or when the line gets over three people long, you know, no one else comes in the door. You have to recreate those. Metrics, especially as your business grows in there, more and more proxies between you and the customer.

Sometimes they're internal proxies, by the way, sometimes their employees or business intelligence or data warehouses that you have to go get the right data from. And then sometimes you're just not even measuring the right thing, but you have to constantly keep the connection to say, I know what my customer experience last week.

I'm a huge customer myself for if he can be for whatever product or service you're building. So those are a couple of things where you have to weave it into everything you do. You can't just talk about it and say, customer obsession is that next level is just a poster on the wall. It won't happen. Get back to the old adage at Amazon, that good intentions don't work mechanisms do 

Brett Berson: [00:56:07] renewing on this theme.

One of the things that you talked about was good intentions, don't work mechanisms. Do, can you give a couple other examples of what those mechanisms or processes look like? With that customer obsession or orientation, or what enables that? So one 

Colin Bryar: [00:56:22] of them is called the correction of errors report where, whether it's a technical glitch that happens on the website or some other type of event in the analog world for these types of errors, the team is responsible to come up with a correction of errors report.

And that's just a very clinical study about, okay, what happened? And we use what's called the five why's approach, you know, borrowed from. A Toyota, why did this happen? And then, okay, well that happened because of X and you go down to the next level of detail and you keep going and going until you've identified all of the root causes, because usually, especially for mature businesses, it's not just one thing that caused an error.

It's usually a sequence of events that really led to a, on big outage or, you know, a shortage in an inventory or a pricing error or whatever it happens is to be another one that it's called voice of the customer. And it's more anecdotes or voice of the seller where you just take a look at a particularly gruesome experience that happened with the customer.

And then at these weekly business reviews, occasionally. Some of them will get up. And sometimes it's reading the actual customer feedback on what happened. They're painful to hear, and you realize you have this connection that here's what my team caused to this customer's lives. And so how am I going to go fix it?

And so these are problems often. They won't bubble up to the top, but, um, Amazon that you don't let those things go. You actually have to go research them and correct them. 

Bill Carr: [00:57:56] Yeah. And to just add onto that last thing, when you dig into that, there are things that happen to customers that really affect their lives in material ways where you can lose so much trust.

Like there've been issues where a customer may have been charged on their credit card for some expensive item, the order was canceled, but there was some issue with their credit card company and there's then a hold still sort of pending on their account and they can't buy something else that they need could be food.

So it's a big deal. And so fixing those issues is real. The third process is one called an Encore, and this process was actually lifted a druggie from Toyota, which Jeff and the leadership team of Amazon looked at Toyota for a lot of inspiration because they're known for quality and figuring out how to reduce defects.

And so the  on-court process in a Toyota production facilities. I understand it is one where if someone working on that line encounters, the same problem multiple times, Um, the car, they can pull a cord, which stops the entire production line because it's a recurring problem. Then the management has to come down, look at what the problem is, and until they can fix this problem, because it's happening, you know, on a recurring basis, they don't restart the line.

And the way that Amazon implemented this is that it often happened that in fact, Jeff encountered this, frankly, by sitting in with a customer support rep and listening to their calls, and the call came in from a customer with an issue on a specific item. And very quickly the customer said, oh, based on this item, I know what the problem is going to be.

And after the call, Jeff asked, how did you know what the problem is going to be? And she said, well, on this particular item, we've seen the same problem over and over and over again, it was some defect about the item itself from the manufacturer. And so Jeff used that then give the power to CS reps to be able to pull the end on cord and make the, when they see some product on Amazon has, this has a recurring problem.

They can pull the cord and that item is no longer available to sell on the detail page and that results in a process for management in the retail organization to scramble and under in, they have to go in and find out what's wrong with this product. Go back with the manufacturer to figure out how to fix it.

And until there's a good solution in inventory that doesn't have this defect is back in our fulfillment center. That item is not for sale. That's a pretty big deal. 

Brett Berson: [01:00:22] Can you handle the concept of customer obsession when you have customers that may be, aren't always aligned. Meaning, for example, you have a seller is one customer at Amazon and you have a buyer.

And at times there's probably examples of where they're not aligned, right? In some ways the customer would prefer to always have a lower price. Maybe a seller would always like a higher price. How do you manage through the multi customer problem? 

Bill Carr: [01:00:51] Well, that's a really good question. And I think that ultimately this is where diving into the details and coming up with specific responses is an important thing.

Let me give you a different example, which is that in my role, I had business partners, you know, the motion picture studios, the record companies. And I would be with these people all the time. I had a real relationships with them and was responsible for making sure that we had terms and relationships. So.

Many times they would come to me with a specific ask or request or policy that I felt wasn't in the best interest of our customers. And a big part of my job, frankly, was to say no to them and make them quite unhappy. And I even remember one of the senior leaders of these companies saying, you know, bill, I really respect you and Amazon, but it would be nice if from time to time when I asked you for something that you could just do it for me as a favor, because my job was really to fight for the customer.

First. No doubt. The example you're giving with a seller is a little more complex, but I would say as a broad generalization, Amazon would favor the end consumer. First, 

Colin Bryar: [01:02:02] the only thing I would add to that is some of these policies, they are tricky to go through, but whatever policy you come up with, You should be okay if it gets out into the world.

So for instance, with this third-party, uh, issue, there was a huge issue with, did customers actually trust third-party sellers? Would the goods be delivered? And then Amazon eventually just came up with a third-party guarantee. They stood behind every purchase. They would make the buyer right with the experience, and then they'd go work with the seller and, you know, take the appropriate steps offline.

And so that was the policy and everyone understood the buyers and sellers involved with it. Understood what was going on. So whatever internal policies you have, if they're big enough and they're customer facing. And they're probably going to get out anyway and, and to, into the press or customers will know.

So make sure that it's a fair policy then, and that reasonable people could look at it and say, oh yeah, I understand why this is happening. That's why Amazon or the company X is doing that. But just to be also cognizant of what is the problem you're trying to, to solve early on in Amazon stays about publishers and didn't want negative customer reviews for, there were only books at the time and they called up Jeff and said, well, you know, you do realize you're in the business of selling books.

Why do you want customers telling people not to buy a book? And Jeff thought about that and he said, well, are we really in the business of selling books? And the answer was no. Amazon was in the business of allowing customers to make informed purchase decisions. And that if in some cases that you steer a customer away from making a purchase, that they would regret later, That occurs trust in that it's beneficial for Amazon in the, in the long haul.

So those are the two things, make sure your policies are transparent and then make sure you're absolutely crystal clear on what types of problems and issues you're trying to adjudicate your 

Bill Carr: [01:04:09] example with where there's a two-sided marketplace. Like for example, Airbnb has had to deal with these big issues, right?

When COVID hit, they did things like cancel a bunch of reservations and give refunds back to the customers who would be renting the houses. And there was an uproar from the people who were homeowners. And I think, you know, if I had to come up with a simple rule, it's the ultimate customer is the one who's sending you the money, not the one where maybe you're paying out a commission, but there's no doubt that there are challenging.

Policy and balancing act decisions to make, whenever you're managing a two-sided marketplace, 

Brett Berson: [01:04:45] I thought we would end by talking a little bit about hiring something that, that both of you mentioned through our conversation and maybe one place to start might be, if I were to look at somebody who we're interviewing and I were to be a fly on the wall and watch the interview, what was going on in that interview?

Why is it structured that way? What are the pieces of it? And what are you trying to learn to ultimately decide whether a given individual should join your team or not? 

Bill Carr: [01:05:11] So the approach at Amazon for interviewing is to focus on it as a data gathering exercise and to structure it so that each interviewer, and typically there are.

About six is actually assigned specific data to collect. And more specifically, they're assigned two or three of the 14 Amazon leadership principles. You split up the 14 across the six interviewers and say, okay, you take these two, you take these three, et cetera, et cetera. So when you're conducting that interview and if you've been assigned, dive deep and.

Customer obsession, then all the questions you're going to ask are really designed to get at whether you believe this candidate will meet or exceed the bar on those leadership principles and you'll do it based on using behavioral interviewing. Meaning, give me an example of a time when, where you're soliciting examples from a candidates past work experience that demonstrate their use or their competency in that leadership.

Sure. Principal, I 

Colin Bryar: [01:06:19] think one follow up. If you were to be a fly on the wall of an Amazon interviewer before and after the interview, if there are lots of interesting things that happened there that are a bit atypical, one is going into the interview. You have a very deliberate job to do. You need to prepare for that.

You can't wing it and you have information or data that you need to collect after the interview ends. Before you talk to anyone else, you have to go write up your feedback again, getting back to written narrative type feedback, and you have to put your vote in before knowing what anyone else has to do voted on that candidate.

And then there's a debrief meeting after that, after everyone's done that, where you're going to go read that feedback and everyone's feedback and everyone's vote and then have the discussion. And at that point, those debriefs, you're not only making the decision on whether to move forward with the candidate, but it's also the feedback loop for interviewing.

You get to see how did this other interviewer on the loop, get this information? What are they doing well that I can learn from? Or what can I teach other people about how to be a better interviewer to, to gather the data that they're supposed to. And then there's of course, the bar raiser, which is an independent person, who's there to guide them process.

Brett Berson: [01:07:36] Are you voting on, would you hire the person or did they exemplify the principles? Both. But then if you're looking at two leadership principles, it would seem to be hard, lacking so much context, whether you would actually hire the person because you're looking at one of the seven dimensions, 10 dimensions that you would ultimately make a hire.

No hire 

Colin Bryar: [01:07:57] your vote is by definition, based on incomplete information, you need to rely on other people on the interview loop, essentially to do their jobs too, to have a complete picture of the candidate. Right. 

Bill Carr: [01:08:10] So when you get to that debrief meeting and then you've got two or three of the 14, then you read the assessment from the other candidates.

Now you have all 14 and now you make a new vote on whether to hire or not hire. And what is unusual for people is that they're used to a, having to try to get it right based on their gut in the interview and not basing it on data and B the idea that they would change their vote after being presented with.

Other feedback and data is also unusual, especially for the hiring manager themselves. So a hiring manager from an outside company would come in and think that their job, if they thought they should hire the person based on the interview, they're going to come into there and try to sell the rest of the people on why we should hire the person, which is exactly the opposite of what you want to do.

In fact, it is a, is now a, uh, an opportunity to complete your understanding of the data, gathered for the candidate, and now make a database decision about whether to hire or not hire the person. 

Brett Berson: [01:09:10] When you think about the mistakes that you made, maybe throughout your journey at Amazon, in terms of hiring.

So people that didn't work out or you had to let go, are there. Specific lessons or ways that you evolved the way that you interview or sort of updating your own mental model or process, or you can just never be perfect. There's not that much to learn. Well, I 

Colin Bryar: [01:09:32] think you can never be perfect because you, by definition only have a couple of hours of time to really assess how someone's going to perform over a number of years.

You're going to change. They're going to change. And so it's a hard thing to get, right. Again, this is one of the things where you want to try to stack the odds as much in your favor as possible. What I've noticed is whenever. There's a team on the fence. You're better off just not hiring and continuing to look because really what's happening there is you're letting different biases creep into that decision.

And you don't really know it at that point, whether it's urgency bias, we're not going to hit our goal. And thus, we hired these five people by the end of the quarter. And so you're probably better off trusting your instincts. And if you're belaboring it, it's probably best just to go look for someone. 

Bill Carr: [01:10:18] So I've made more than my fair share of hiring mistakes during my time at Amazon and afterwards too.

And. Not only was I a bar raiser at Amazon, but I was a member of the bars or core, which means I was a process expert and had conducted nearly a thousand interviews during my time in 15 years. But when I look at each hiring mistake that I made, if I go back and look at the process and the interview loop, I can always see where I slash.

We made the error. And generally speaking, it was, I, as the hiring manager did not focus on the process. Like I should have either. I was focused more on selling the candidate than actually deeply interviewing them, or I'd ignored specific feedback from others on the loop, because in many cases, even in debrief, it can be a jump ball.

And the data that was gathered that said why this person was going to work out. Was there. But I chose to believe that that would not hold the person back. So generally speaking, in my experience, the Amazon process done well, we'll collect the data that tells you why this person isn't going to work out.

It's a question of how closely you pay attention to that signal. 

Brett Berson: [01:11:26] Great. Well, thank you both so much for spending this time with me. I have pages of other questions. So maybe in the future you can come back on, but this was such a delight. Thank you so much. And congrats 

Colin Bryar: [01:11:37] on shipping the book. 

Bill Carr: [01:11:38] You very much.

Thank you. Really enjoyed it.