Operations vs. Algorithms: Advice for scaling startups from Ian Wong, CTO of Opendoor

 So I thought we could break the conversation up into a few different chunks. One is we could kind of start by talking about some, the things you figured out as it relates to data science, then talk about some of the operational things that you figured out in terms of scaling open door.

then we can end on sort of talking about the different parts of what it means to be, CTO across different phases of a company's life. Um, and so kind of in this first category, I thought we could sort of start a little bit broad and maybe you could share what you figured out or what you've noticed that most startups or maybe scaling companies get wrong as it relates to data science.

 Yeah. I think data science, uh, can mean a lot of things to different companies. Um, in my experience, both at square and, uh, at open door, really the. A core problem underlying the business was a data science one. And so one of the, um, key items that I believe a business has to figure out is what are the key risks that a business has?

What are the things that can prevent the company from realizing its vision? And then see if data science is a part of that. And if so, you know, we can red chat through how to hire for that talent or how do we integrate data science into the business. And so I think, you know, one of the first things that a company has to really think through is what are the key risks that the business has in order to realize its vision.

And with part of that, um, falls under the bucket of data science.

 So in Square's case, right, what is the premise of square? It is, at least in the beginning was about giving people the ability to accept payments very easily, right? So it started by building these, uh, white, small cart readers that they mail to everybody. And you can simply attach that cart reader to your phone.

 And you can literally go from, sign up to a cart swipe to having that payment land in your account all in about 10 minutes. So that was really the magic behind square. And if you break that problem down and ask, Hey, what are the components to deliver that vision of making it easy to accept payments while you need to have great user experience, you need to have great backend engineering because obviously the security concerns involved with accepting payments.

Um, and also the distributed systems concerns in scaling. The acceptance of payments is huge. And then of course, risk. you're giving out all these cart readers and enabling so much activity that it's gonna be bad activity as part of that too. And so how do you make sure that the ecosystem is safe and sound?

And so if you think of that third pillar risk, this is obviously where a data science really comes into play. Um, and open doors case. What we wanna do, um, over time is to be a one stop shop for all things for Allstate, where it is seamless and frictionless to transact, where we started was helping homeowners be able to sell their homes much more easily, right?

Selling a home is a super painful process. First of all, as the single largest asset that most people have. And if you were to sell it, you have to go through traditionally a nine 80 day process of uncertainty and stress. And what we wanna do is do what is, has been available in many other asset classes.

Uh, which is to say, to be able to sell it right away to a trusted counterparty. And if you break that problem down where we are giving sellers the ability to sell homes nearly instantly, we have to have great customer experience, great operations, the ability to raise capital, um, IE, having a strong capital markets function and last but not least, certainly not least having a really robust pricing and data function so that we know what is a competitive offer that we can, um, make to sellers.

And how do we, uh, resell that home? Um, eventually. So if you peel back again, the capabilities that the companies need to have both across square and open door, you'll find out that data science is a core component of that.

 So given how important that was to the foundational bets of open door, how did that translate into how you approach data science in like the first 90 days or three or four months of building open door?

In the early days, the way we thought about building a company, and that really has stayed with us since it's about identifying the key risks to the business and what are the solutions, or at least the approach to reduce that risk. And so, as it relates to data science, um, again, we have to really be great at pricing houses, right?

 How do we price how much we should acquire it at? And also, how do we think about reselling the home? How do we think about inventory management and risk management more broadly? And so we basically set out these milestones around data collection around. Creating our first set of algorithms with a certain level of accuracy and also around setting up, not just the data science side of the house, but the operation side of the house, because those are really turbines that work together to help build your engine around, uh, valuing homes.

So how do we construct this, uh, business process or business capability that combines both data science, engineering, operations, and other disciplines in order to actually, um, value homes, competitively, uh, so that we have a business that's durable. So those are some of the things that we did in the early days to really set that up.

And of course, you know, I think there's a lot that goes into how do we construct the early team and how do we think about the early hires, early data scientists that join the team as well?

 guess before we talk about sort of the, your point, the early team, that sort of the action on this, can you talk in a little bit more detail in terms of the actual work or roadmap that you established and maybe how you figured that out? Um, and maybe how that might be different than how other people thought about the data problem in terms of in this case, maybe pricing homes.

 of all of folks, when they think about, um, open door and at least in the early days, I think now the, the understanding has evolved a lot, a lot in the early days, folks thought, Hey, you know, you just gotta build a home valuation model, you know, similar to what some of the other, uh, internet real estate companies have done.

 And the answer is, yes, we have to build that and we have to build a lot more, right. So it's not just, Hey, how much do we think a home is worth today? We also have to understand. How much do we think a home is worth three months from now, which is typically how long it takes to sell a house. And in some cases, six months from now to help us figure out what the risk might look like.

Uh, in the long term, we also have to think about how should we, um, construct an offer for the customer and also how do we resell a house, right? Cause when you're in an inventory business, reselling that inventory it's a really important, uh, set of pricing decisions that have relationships with how we buy the home, but have this own distinct set of challenges.

And so overall I'd say one of the things that folks might misunderstand open door in early days is they think of open door as having just one evaluation problem, but really it's a series of evaluation problems and a series of risk and asset management problems. And what we're trying to construct is actually a holistic asset management, uh, system.

So that's something that I think, um, is, is a little bit different. And one more thing I would add to that is, um, again, when folks think about data science, they think of like, you know, former PhDs or they think about statisticians or machine learning folks, if you are truly trying to build a data science capability, it's not just the algorithms that we're thinking through.

You also have to think about the data assets and also the operational arm of that capability, because all of that have to work in sync for you to have a really strong data science function.

 data science function side, what did the early product roadmap look like? as you describe sort of the problem space, it, it seems like such a big, hard, difficult problem. And so how did you pull it apart into its component parts that you and the team could actually start going after?

 So generally speaking, I'm a firm believer in going end to end early, um, and having a scaffold of an end to end process that you can then make more sophisticated over time. Uh, this is a bit of a crude analogy because it kind of came from, uh, you know, the war time. But it's an analogy that I picked up that I really like.

And I learned it from a book, uh, called the pragmatic programmer, one of the best books ever. about software engineering and the terms called tracer bullet. And the idea is you want to use, um, cheap. Cheaper, um, kind of artillery to see if you actually nail the target or you're kind of roughly where you want to land before you load up the more expensive artillery.

And I, I think of that, uh, I, I like the analogy a lot when thinking about building businesses and building business capabilities. So what is a low fidelity end to end process look like? Where in many cases you might actually have operations substituting the role of a data science, um, algorithm or a data science function, but then you kinda lay that end to end.

So the first thing is, okay, what does that end to end valid delivery system look like? So how do we price a home up front? How do we think about reselling a house? How do we think about, um, managing a portfolio of, and homes. You might do that fairly manually initially, but then you lay the groundwork in a scaffolding and the framework so that you can start to bit by bit replace some of these pieces, um, with more sophisticated algorithms.

Right. And, and actually as we evolve, you know, like entire parts of the system might look very different because your technology curve has matured a lot more, but in the early days it was okay, how do we set the end to end? Like, what does the system look like and how do we start to lay the roadmap of the data that's being generated or the decisions that could be started to be automated or at least guided by, um, data science and algorithms.

 So could you, when you think back to the early days, can you talk about, maybe tell the story or zoom all the way in, on of this more crew to end to end process and maybe one of the component parts and how you built on top of that over time and maybe landed on an end state that it is today.

 one of the components of the evaluation pipeline or the asset management pipeline at large, um, is how do we value home upfront? Um, and one of the things that as both common across square and open door is Pandora is you might lean on, uh, human experts in the beginning, but you wanna make sure that's done in a way that gives you structured data and gives you the ability for your algorithms to plug in easily.

 And the way you commonly do that is by actually having very tight control of the tools that the operators are using to conduct that decision, or to make that decision. In Square's case, it is reviewing an account to see if we can actually settle, um, the amount that's owed to the merchants, uh, at open doors case it's well, how do you.

Value a house. And so in both cases, we've devoted a lot of energy in building the tools, you know, in the former case, it is the account review tool and the latter case with open door. It is the valuation tool for our house. And we spent a lot of energy actually making sure that we are tightly controlling what the UI looks like and the data that feeds into that and the data that's emitted from that.

Right? Because I think one of the most important aspects of building a data science function is thinking about the data mode that you're generating over time. And that data mode isn't just by consolidating third party data. Of course that's important, but it's also about figuring out what is the unique proprietary first party data that is extremely important to curate and accumulate over over time.

Um, so the valuation tool for instance, is something that we built that has really tight control. And again, like the operator used to initially. Then over time, what we did was, well, it, it gave our algorithms a lot of insights as to where it might be off. And then over time we, uh, guided the valuations and then over time we actually automated, uh, a lot of it.

So that's kind of like one example of how we might go up that, uh, maturity curve.

 On that point. I think one of the things that I've observed is there are so many companies who sort of begin with some human in the loop operator, sort of human component. they say in X number of years, we're gonna automate this and you check in an X number of years  it effectively never gets automated or something doesn't work, or they're not able to do it.

And so they don't get that leverage over time. Is there anything you did or you reflect on that allowed you to do that so successfully.

 Yeah, I think it's a really hard problem. And I, I totally understand where you're coming from and I, I see this, um, a fair amount as well. Um, because the reality is on a marginal cost basis, it is easier to scale with human beings, right? So like, if you are just, um, doing greedy search, so to speak and say, okay, if I could deploy a marginal dollar and maximize that marginal dollars return, um, it is cheaper and faster to hire an operational staff to attend to the low volumes of decisions that you have to make offers that we have to make an open door or the reviews that need to happen at squares case.

Um, and it's cheaper and faster to do that than it is to hire and build up an entire pricing data capability for open door, right? And there are really sharp folks, really sharp operators that you can find, and they will do a great job. And they're awesome. The issue is if you lean purely on operations and humans, it will yield just a kind of scale.

And you're not gonna be able to get the level of precision and scalability that the company will need. And so I think it actually, first and foremost, start with almost like an article of faith that the executive team needs to have like conviction that at open door. If we truly want to deliver competitive offers at scale, in such a way that enables open door to be durable, then we need to build an asset management system that combines the best of operations with the best updated science.

Right. So I think it has to start with number one conviction. And I, I emphasize this point because when I see cases, when I see cases where you're constantly doing this marginal cost calculation and therefore not being able to justify their investment in data science, that actually shows a lack of conviction.

So first and foremost, like, you know, as the executive team and the founding team actually convicted in the value of data science, that's that's number one, um, And then once you are, you know, once you are aligned on the why, then you have to really figure out the how, right. So there is a lot of techniques, you know, some of which I mentioned, like, how do you set the feedback loop between operations and data science, right?

Like how do you actually, um, paint a picture of the maturity curve so that both the operations team and the data science team are bought in and you kind of have to set a few pretty aggressive milestones for data science, right? You, it's never gonna be fully ready when you deploy it. And so you have to figure out, okay, well, When is the data science team gonna guide certain decisions or better yet?

When are, when is the data science team going to start to automate certain decisions? How are we gonna measure the accuracy or, you know, the fitness of those decisions and how do we have a graceful degradation? So, you know, if certain algorithms, um, aren't up to par, how do you dial up the proportion of decisions that are made by machines versus the humans, but you have to have actually pretty aggressive deadlines and thresholds, meaning, you know, in some cases.

An algorithm doesn't have to be strictly better than a human being for you to roll it into production. It just has to be no worse or better yet. It just has to be so that, you know, for subset of decisions, it is no worse. So you wanna find these pockets where you start to put data sciences through front lines so that the data scientists themselves sort of feel the heat.

Um, and I think that's really important cuz that's real skin in the game for data scientists and they like that. And once you're able to have these air pockets where they are really showing the value of a predictive system that they've been developing, I think the, you know, there's the natural progression to expand that.

 that point. Are there other ways that you've, uh, organized or run the team that you think had an outsized impact? Or maybe you think about the way that you ran and run today, is there some unique things that you figured out maybe relative to the way that other folks operate and execute their data science functions?

 I'd say there are a few things that were really important to me, uh, especially with the early data science team. Um, number one is, and this kind of relates to this types of data scientist that you wanna bring on in the early team. One is I think having a real domain understanding is really important.

And so what I mean by that is. I think of data science, actually, a lot of times what a data scientist does is that it generalizes human intuition into data and algorithms. And then you subject those intuitions to a back test to see which of these intuitions are actually generalizable, but the core, idea behind that statement is intuition, right?

Like, so a data scientist has to have a really good understanding of the domain, so okay. To make it more concrete, you know, when I was at square, one of the best, uh, things I did there was I actually, um, mocked into issues of our operators and I caught up, uh, potential fraudsters. Right. And it was a really mind blowing experience cuz like, you know, you are seeing this whole GAT of people who, who are real merchants and pretend merchants and you know, some of those conversations are, uh, quite colorful.

Um, in the case of open door, Uh, go in valley homes, right? And so you are sitting, uh, you're sitting alongside, um, folks in our pricing operations team. You're going out to the field to see the houses. Um, you're sitting next to our homes team and figuring out, um, how to think about renovations, right?

And you're on the hook for actually doing some number of valuations, uh, when you're onboarded as a data scientist here. And so these are some of the ways in which you can really develop domain understanding. And I think that is super important because you can't have an ivory tower, um, you know, in data science,

 I I'm just so curious, uh, about the fraudster conversations. Are, are there are story or stories that come to mind in terms of other than I'm sort some bizarre things that happened, but sort of maybe how, how that work translated back into what you were building.

 generally speaking, I I'd say, um, you can pick up on some signals that are helpful at large and also some tools enable operators to do more efficient account review. And so for instance, um, in credit card fraud, uh, cart, not presence, meaning, you know, a payment that was entered via, you know, via manual entry.

Those are, have a higher risk of fraud than cart presence, meaning the cart wasn't there and swiped for a transaction. And so when you see a lot of car, not presence for a, you know, for high dollar amounts for, um, uh, transactions that are supposed to be for travel at a later date or concerts for a later date, you start to question why, um, and you start to be able to piece together.

You know, some of the signals are very relevant, like. Okay. Well, if you're a, uh, travel agencies that are booking, you know, multi thousand dollar travel, uh, bookings that are for a very much later date, all car not present, and you're seeing some amount of fraud, like you talk to these people and they have a very, you know, thin story around how their business is constructed.

You, you have greater fiction that, uh, you know, these are some of the signals that are promising or account review. Like, you know, how do you, uh, understand for an, uh, for an operator, the types of questions that they ask and how do you actually make those questions, uh, show up on the tool. So it's guided.

And so a lot of those practice also apply to open door, right? Like what are some of the things that, uh, you look at from a data element standpoint, when you value home that might be gotchas, right? Like things that might have an outsized impact on the value of the home one way or another, how do you start to, uh, generate those elements in the UI, in the tools and capture that data so that it informs her algorithms as well.

 So something that you touched on a little bit, as we've been talking is, um, is the early team in the data science function. So I'm excited to hear more about what did that look like for open door? Maybe? Why did you structure it that way maybe what does that talent look like in the early days?

 yeah, that's a great question. And then absolutely changes over time. Um, but before talking about, you know, what, what changes I think, as we think about an early data science team in a startup. Let's think about the context right in this early phase. Number one is your data science work band is really immature.

Meaning your engineers are probably more focused on shipping the products and you might have some data engineers as the case with open door, but they're probably gonna be focused on data ingestion, entity of resolution, right? Like for us at open door, real estate, data is messy. Uh, and the sources of data is very fragmented between recorder, offices, assessor offices, MLSs, and other types of, um, geospatial data sets.

So our data engineers are just really focus on just pulling all that data set together. And so again, the data science work bench is immature, um, right. And that's number one. Number two is the problem is not well defined. And in some cases the data assets aren't even defined and you have to go out to get the data or work with engineers to specify how the data will be collected or cleaned up.

So your work isn't gonna be specked out. And in fact, your job is to spec out the problem. Your job is to spec out the objective function and that's not given to you. And the third is, um, the company's evolving very quickly, right? And so you have different teams of different types of data science problems.

And so there's a, you know, it's an amorphous blob in the early days in some, in some ways. And so if you think about those, that context, the early startup data scientists, I think have to have a few really important attributes. Number one is I think first and more foremost, they have to be a self starter.

Um, and I think that actually filters out a lot of data scientists, unfortunately, because I think in a larger company, there's just so much affordance and so much, um, infrastructure that you can rely, rely on. And then the absence of all that affordance and all that infrastructure, it's hard for some of these individuals to be productive.

So number one, just being a self starter, number two. And I think this is the part that gets a lot of data scientists who might come out of academia is that you have to be a sponge and you have to be deeply curious about the business. Um, you know, it's, it's easy to fall in love with the method, especially coming from a academic background, but what's more important in, in industries, falling in love with the customer and falling in love with the business problem.

Um, and so generally I call these folks that we were trying to look for in the early days T-shaped individuals where there's an area of expertise, right? Like data, data, science, machine learning, statistics, what have you. But that top of their skill set is wide enough so that they can collaborate effectively with, um, other functions.

And they're curious about the other functions and they have enough of a skillset to be able to unblock themselves.

 So when you think about these kind of two big characteristics, self starter sponge, did that map to where you looked for these people or what the ideal background was? I.

 generally speaking, I think a lot of great, um, early data scientists or data scientists are suitable for early stage startups. They either have had experiences in other early stage startups. Um, they might be early in their careers or maybe even in grad school, but have had shown a lot of independence and autonomy to try things on their own or to, you know, participate in cargo competitions.

 And in some cases it could be in larger companies, but again, you have to have a stronger filter for whether or not they are able to bootstrap themselves. Um, but as we think of thought about the early team At open door, you know, I was able to pull from my network from, from square and from some of the, uh, other, uh, networks that we had in the founding team, like Eric's network from Trulia, um, and JDS network from volunteers.

So we're able to get folks that have really valuable early stage tech experience. And I think that was really, really helpful for us.

 For the folks that you didn't work directly with, what was the interview or evaluation process and maybe how did that map back to of those two big characteristics and, and, you know, said differently if, starting is critical for a data scientist to have success in the early days at open door, what were you doing before you decided to hire them to get to conviction that they would sort of spike in that area.

 Yeah, I spent quite a bit of time thinking through that, especially in the early days, because you know, in a larger company, you have a pretty much standardized set of questions that you ask. And these questions tend to be kinda like case studies or, you know, things that are not, to be honest, like directly relevant to the work that you're trying to do.

And in the early startup phase, like you can't afford that. Like, you know, when you're bringing the person they're gonna work on the thing that needs to be done. And so you all, like my philosophy was how do I mirror the interview process to mimic what they need to be doing on the job as closely as possible.

So I know they'll be great at, you know, on day of one. And so the interview process was literal. Literally I took real estate data, um, from the many sources that I, uh, we have. For, uh, you know, for data lessons and purposes, I like actually spent some time randomizing, some of that data, Deon, uh, anonymizing, some of that data, but generally speaking, you're working with for all intensive purposes, the data set that we are working with.

And then I ask you to build a model. I ask you to actually like build a model within two or three hours. And the idea is, again, that tracer bullet mentality, like, can you go end to end? Like, where are you getting stuck? And I'm, I'm sitting with you, like I'm spending hours and hours to these candidates.

I'm sitting side by side to make sure that if there's something trivial, um, that they're stuck on, like I'm here to unblock cuz in real life, I am here to unblock you. Right? So we're, we are gonna be doing a lot of pair programming together. And so I, I basically simulated as much as possible what the early stages of opener looked like, which was working with a lot of.

Fragmented fragmented data sets having to boost strap your tool chain, meaning there isn't like, you know, this nice Google infrastructure. That's gonna allow you to, you know, ship models and, and, uh, using their, their software. Like you have to figure out how to use a lot open source. Um, and we just kind of worked through the problem together.

So that's, that's how I made sure that the candidates were being assessed, um, in a way that mimics what they have to do, uh, in production. Yeah.

 Can you think back to any hiring errors that you made in the early days where you had to ask someone to depart and was there a thread that tied any of those early mistakes together? The thing that you sort of got wrong in this evaluation process?

 I mean, I, I think generally speaking what's really important in the early days is again, the self starting, um, capability. Um, and that's actually in many cases, much more important than existing domain expertise. And so what I mean by that is you might have folks that have many, many years of experience in a related field or I saw this at square two, like, uh, payments or an open doors case real estate, and that experience is valuable, but there are other ways to come up to speed without experience very quickly.

And I'd rather over index on your ability to unblock yourself and your ability to get, uh, up to speed very quickly. One, one phrase I love from the professor at Stanford was, um, a bit of slope mix up for a lot of intercept, and I think that's very true. Um, and so I think where we've made hiring mistakes, especially early on, was calling in quote, unquote, the specialists or the experience hires.

Uh, a bit too early, uh, and if they don't have, or they lost a skillset to be that self starter, I think it actually can create, um, quite a bit of problems.

 As we've been talking about the, the early days and, and some of the recruiting ideas at, at open door, I'm interested just more broadly given the, the folks that were around the table with Keith and Eric JD and you, what, have you learned or figured out as it relates to culture? Because I think that's, that's a, a big idea that powers all of your work.

And I think you're all quite opinionated about it and maybe approach it a little bit differently than some other companies. And so what, what, what have you kind of figured out or learned about culture?

 Certainly we had and certainly we had an opinion set of founders that's for sure. Between Keith, Eric, JD, and myself, that's for sure. And, you know, not all of our opinions are always the same. Um, but I, I will say one thing that I, I think we learned is, um, It's important to actually name the values, uh, because they could be very implicit and sometimes where you have clashes and decisions, it could be a result of having misalignment and values and actually having a discussion around what the values mean is really important.

So for instance, um, build openness is one of the, our, our earliest values and continues to be our value today. What does that mean? Right. And like, you know, uh, how do you actually practice it? It's easy to practice Val practice values when things are going well, right? Like when everything's up to enter, right?

Like it's easy to share, like, Hey, here's how many contracts we're signing every, every week. Here's the number of hires we're making. Here's how, how much, how many dollars are flowing through the system? That's easy. Uh, valleys are tested when, well, shit like, you know, we haven't had a contract in a few weeks, you know, in the early days, I still remember. Having convinced great engineers to leave the apples and Googles and squares of the world to join open door. And like we had a really slow winter and folks asked me one by one in our one on ones, Hey, what's happening with the business? And I think these are the moments where, um, build openness matters, right?

Like being honest and about where the business is, what the opportunities are, but what the risks are. And that's how you can really enroll, um, you know, adults and really talented create creatives to be part of your journey. Um, so I think being intentional about culture, being able to specify what your values are, um, really, really matters and actually translate those values into rituals.

Be it again, like, you know, the one-on-ones, but also like the all hands, the retros, um, you know, how we share our, our board decks, especially in early, early days, like all, all that are really important.

 can you walk through some of the early values and, and maybe why you all landed on them?

62f413f1f0fe651234ed9454

Ian Wong is the co-founder and CTO of Opendoor. He discusses the tricky balance between operations and data science in the early days of the startup, and his advice for other founders looking to make data science core to their business. 

Operations vs. Algorithms: Ian Wong's advice for scaling startups

Opendoor's Ian Wong wrestles with the tricky balance between scaling with operations and data science, with tons of advice for early-stage founders.