Operations vs. Algorithms: Advice for scaling startups, from Opendoor CTO Ian Wong

[00:00:00] Brett Berson: Ian, thank you so much for joining us.

[00:00:02] Ian Wong: glad to be

[00:00:04] Brett Berson: So I thought we could break the conversation up into a few different chunks. One is we could kind of start by talking about some, the things you figured out as it relates to data science, then talk about some of the operational things that you figured out in terms of scaling open door.

[00:00:19] And then we can end on sort of talking about the different parts of what it means to be, uh, CTO across different phases of a company's life. Um, and so kind of in this first category, I thought  we could sort of start  a little bit broad and maybe you could share  what you figured out or what you've noticed that most startups or maybe scaling companies get wrong as it relates  to data science.

[00:00:43] Ian Wong: Yeah. I think data science, uh, can mean a lot of things to different companies. Um, in my experience, both at square and, uh, at open door, really the. A core problem underlying the business was a data science one. [00:01:00] And so one of the, um, key items that I believe a business has to figure out is what are the key risks that a business has?

[00:01:11] What are the things that can prevent the company from realizing its vision? And then see if data science is a part of that. And if so, you know, we can red chat through how to hire for that talent or how do we integrate data science into the business. And so I think, you know, one of the first things that a company has to really think through is what are the key risks that the business has in order to realize its vision.

[00:01:34] And with part of that, um, falls under the bucket of data science. 

[00:01:41] Ian Wong:  So in Square's case, right, what is the premise of square? It is, at least in the beginning was about giving people the ability to accept payments very easily, right? So it started by building these, uh, white, small cart readers that they mail to everybody. And you can simply attach that cart reader to your [00:02:00] phone.

[00:02:00] And you can literally  go from, sign up to a cart swipe to having that payment land in your account all in about 10 minutes. So that was really the magic behind square. And if you  break that problem down and ask, Hey, what are the components to deliver that vision of making it easy to accept payments while you need to have great user experience, you need to have great backend engineering because obviously the security concerns involved with accepting payments.

[00:02:26] Um, and also the distributed systems concerns in scaling. The acceptance of payments is huge. And then of course, risk.  you're giving out all these cart readers and enabling so much activity that  it's gonna be bad activity as part of that too. And so how do you make sure that the ecosystem is safe and sound?

[00:02:44] And so if you think of that third pillar risk, this is obviously where a data science really comes into play. Um, and open doors case. What we wanna do, um, over time is to be a one stop shop for all things for Allstate, [00:03:00] where it is seamless and frictionless to transact, where we started was helping homeowners be able to sell their homes much more easily, right?

[00:03:09] Selling a home is a super painful process. First of all, as the single largest asset that most people have. And if you were to sell it, you have to go through traditionally a nine 80 day process of uncertainty and stress. And what we wanna do is do what is, has been available in many other asset classes.

[00:03:28] Uh, which is to say, to be able to sell it right away to a trusted counterparty. And if you break that problem down where we are giving sellers the ability to sell homes nearly instantly, we have to have great customer experience, great operations, the ability to raise capital, um, IE, having a strong capital markets function and last but not least, certainly not least having a really robust pricing and data function so that we know what is a competitive offer that we can, [00:04:00] um, make to sellers.

[00:04:02] And how do we, uh, resell that home? Um, eventually. So if you peel back again, the capabilities that the companies need to have both across square and open door, you'll find out that data science is a core component of that.

[00:04:18] Brett Berson: So given how important that was to  the foundational bets of open door, how did that translate into how you approach data science in like the first 90 days or three or four months  of building open door?

[00:04:32] Ian Wong: In the early days, the way we thought about building a company, and that really has stayed with us since it's about identifying the key risks to the business and what are the solutions, or at least the approach to reduce that risk. And so, as it relates to data science, um, again, we have to really be great at pricing houses, right?

[00:04:57] How do we price how much we should [00:05:00] acquire it at? And also, how do we think about reselling the home? How do we think about inventory management and risk management more broadly? And so we basically set out these milestones around data collection around. Creating our first set of algorithms with a certain level of accuracy and also around setting up, not just the data science side of the house, but the operation side of the house, because those are really turbines that work together to help build your engine around, uh, valuing homes.

[00:05:33] So how do we construct this, uh, business process or business capability that combines both data science, engineering, operations, and other disciplines in order to actually, um, value homes,  competitively, uh, so that we have a business that's durable. So those are some of the things that we did in the early days to really set that up.

[00:05:52] And of course, you know, I think there's a lot that goes into how do we construct the early team and how do we think about the early hires, [00:06:00] early data scientists that join the team as well?

[00:06:04] Brett Berson: guess before we talk about sort of the, to your point, the early team, that sort of the action on this, can you talk in a little bit more detail in terms of the actual work or roadmap that you established and maybe how you figured that out? Um, and maybe how that might be different than how other people thought about  the data problem in terms  of in this case, maybe pricing homes.

[00:06:30] Ian Wong:  of all of folks, when they think about, um, open door and at least in the early days, I think now the, the understanding has evolved a lot, a lot in the early days, folks thought, Hey, you know, you just gotta build a home valuation model, you know, similar to what some of the other, uh, internet real estate companies have done.

[00:06:51] And the answer is, yes, we have to build that and we have to build a lot more, right. So it's not just, Hey, how much do we think a home is worth [00:07:00] today? We also have to understand. How much do we think a home is worth three months from now, which is typically how long it takes to sell a house. And in some cases, six months from now to help us figure out what the risk might look like.

[00:07:11] Uh, in the long term, we also have to think about how should we, um, construct an offer for the customer and also how do we resell a house, right? Cause when you're in an inventory business, reselling that inventory  it's a really important, uh, set of pricing decisions that have relationships with how we buy the home, but have this own distinct set of challenges.

[00:07:38] And so overall I'd say one of the things that folks  might misunderstand open door in early days is they think of open door as having just one evaluation problem, but really it's a series of evaluation problems and a series of risk and asset management problems. And what we're trying to construct is actually a holistic asset management, uh, system.

[00:07:57] So that's something that I think, um, is, is a little bit different. [00:08:00] And one more thing I would add to that is, um, again, when folks think about data science, they think of like, you know, former PhDs or they think about statisticians or machine learning folks, if you are truly trying to build a data science capability, it's not just the algorithms that we're thinking through.

[00:08:18] You also have to think about the data assets and also the operational arm of that capability, because all of that have to work in sync for you to have a really strong data science function.

[00:08:32] Brett Berson: on the data science function side, what did the early product roadmap look like?  as you describe sort of the problem space, it, it seems like such a big, hard, difficult problem. And so how did you pull it apart into its component parts that you and the team could actually start going after?

[00:08:54] Ian Wong: So generally speaking, I'm a firm believer [00:09:00] in going end to end early, um, and having a scaffold of an end to end process that you can then make more sophisticated over time. Uh, this is a bit of a crude analogy because it kind of came from, uh, you know, the war time. But it's an analogy that I picked up that I really like.

[00:09:17] And I learned it from a book, uh, called the pragmatic programmer, one of the best books ever. about software engineering and the terms called tracer bullet. And the idea is you want to use, um, cheap. Cheaper, um, kind of artillery to see if you actually nail the target or you're kind of roughly where you want to land before you load up the more expensive artillery.

[00:09:40] And I, I think of that, uh, I, I like the analogy a lot when thinking about building businesses and building business capabilities. So what is a low fidelity end to end process look like? Where in many cases you might actually have operations substituting the role of a data science, um, algorithm or a data science function, [00:10:00] but then you kinda lay that end to end.

[00:10:02] So the first thing is, okay, what does that end to end valid delivery system look like? So how do we price a home up front? How do we think about reselling a house? How do we think about, um, managing a portfolio of, and homes. You might do that fairly manually initially, but then you lay the groundwork in a scaffolding and the framework so that you can start to bit by bit replace some of these pieces, um, with more  sophisticated algorithms.

[00:10:29] Right. And, and actually as we evolve, you know,  like entire parts of the system might look very different because your technology curve has matured a lot more, but in the early days it was okay, how do we set the end to end? Like, what does the system look like and  how do we  start to lay the roadmap of the data that's being generated or the decisions that could be started to be automated or at least guided by, um, data science and algorithms.

[00:10:51] Brett Berson: So could you, when you think back to the early days, can you talk about, maybe tell the story or zoom all the way in, [00:11:00] on kind of this more crew to end to end process and maybe one of the component parts  and how you  built on top of that over time and maybe landed on an end state that it is today.

[00:11:11] Ian Wong:  one of the components of the evaluation pipeline or the asset management pipeline at large, um, is how do we value home upfront? Um, and one of the things that as both common across square and open door is Pandora is you might lean on, uh, human experts in the beginning, but you wanna make sure that's done in a way that gives you structured data and gives you the ability for your algorithms to plug in easily.

[00:11:39] And the way you commonly do that is by actually having very tight control of the tools that the operators are using to conduct that decision, or to make that decision. In Square's case, it is reviewing an account to see  if we can actually settle, um, the amount that's owed to the merchants, uh, at open doors case it's [00:12:00] well, how do you.

[00:12:00] Value a house. And so in both cases, we've devoted a lot of energy in building the tools, you know, in the former case, it is the account review tool and the latter case with open door. It is the valuation tool for our house. And we spent a lot of energy actually making sure that we are tightly controlling what the UI looks like and the data that feeds into that and the data that's emitted from that.

[00:12:25] Right? Because I think one of the most important aspects of building a data science function is thinking about the data mode that you're generating over time. And that data mode isn't just by consolidating third party data. Of course that's important, but it's also about figuring out what is the unique proprietary first party data that is extremely important to curate and accumulate over over time.

[00:12:47] Um, so the valuation tool for instance, is something that we built that has really tight control. And again, like the operator used to initially. Then over time, what we did was, well, it, it gave our algorithms a lot of [00:13:00] insights as to where it might be off. And then over time we, uh, guided the valuations and then over time we actually automated, uh, a lot of it.

[00:13:09] So that's kind of like one example of how we might go up that, uh, maturity curve.

[00:13:14] Brett Berson: On that point. I think one of the things that I've observed is there are so many companies who sort of begin with some human in the loop operator, sort of human component. And they say in X number of years, we're gonna automate this and you check in an X number of years  and it effectively never gets automated or something doesn't work, or they're not able to do it.

[00:13:38] And so they don't get that leverage over time. Is there anything  you did or you reflect on that allowed you  to do that so successfully.

[00:13:47] Ian Wong: Yeah, I think it's a really hard problem. And I, I totally understand where you're coming from and I, I see this, um, a fair amount as well. Um, because the reality is on a marginal cost basis, it is easier to [00:14:00] scale with human beings, right? So like, if you are just, um, doing greedy search, so to speak and say, okay, if I could deploy a marginal dollar and maximize that marginal dollars return, um, it is cheaper and faster to hire an operational staff to attend to the low volumes of decisions that you have to make offers that we have to make an open door or the reviews that need to happen at squares case.

[00:14:23] Um, and it's cheaper and faster to do that than it is to hire and build up an entire pricing data capability for open door, right? And there are really sharp folks, really sharp operators that you can find, and they will do a great job. And they're awesome. The issue is if you lean purely on operations and humans, it will yield just a kind of scale.

[00:14:45] And you're not gonna be able to get the level of precision and scalability that the company will need. And so I think it actually, first and foremost, start with almost like an article of faith that the executive team needs to have like conviction that at [00:15:00] open door. If we truly want to deliver competitive offers at scale, in such a way that enables open door to be durable, then we need to build an asset management system that combines the best of operations with the best updated science.

[00:15:14] Right. So I think it has to start with number one conviction. And I, I emphasize this point because when I see cases, when I see cases where you're constantly doing this marginal cost calculation and therefore not being able to justify their investment in data science, that actually shows a lack of conviction.

[00:15:33] So first and foremost, like, you know, as the executive team and the founding team actually convicted in the value of data science, that's that's number one, um, And then once you are, you know, once you are aligned on the why, then you have to really figure out the how, right. So there is a lot of techniques, you know, some of which I mentioned, like, how do you set the feedback loop between operations and data science, right?

[00:15:56] Like how do you actually, um, paint a picture of the [00:16:00] maturity curve so that both the operations team and the data science team are bought in and you kind of have to set a few pretty aggressive milestones for data science, right? You, it's never gonna be fully ready when you deploy it. And so you have to figure out, okay, well, When is the data science team gonna guide certain decisions or better yet?

[00:16:21] When are, when is the data science team going to start to automate certain decisions? How are we gonna measure  the accuracy or, you know, the fitness of those decisions and how do we have a graceful degradation? So, you know, if certain algorithms, um, aren't up to par, how do you dial up the proportion of decisions that are made by machines versus the humans, but  you have to have actually pretty aggressive deadlines and thresholds, meaning, you know, in some cases.

[00:16:50] An algorithm doesn't have to be strictly better than a human being for you to roll it into production. It just has to be no worse or better yet. It [00:17:00] just has to be so that, you know, for subset of decisions, it is no worse. So you wanna find these pockets where you start to put data sciences through front lines so that the data scientists themselves sort of feel the heat.

[00:17:12] Um, and I think that's really important cuz that's real skin in the game for data scientists and they like that. And once you're able to have these air pockets where they are really showing the value of a predictive system that they've been developing, I think the, you know, there's the natural progression to expand that.

[00:17:28] Brett Berson: that point. Are there other ways that you've, uh, organized or run the team that you think had an outsized impact? Or maybe if you think about the way that you  ran and run today, is there some unique things that you figured out maybe relative to the way that other folks operate and execute their data science functions?

[00:17:53] Ian Wong: I'd say there are a few things that were really important to me, uh, especially with the early data science team. Um, number [00:18:00] one is, and this kind of relates to this types of data scientist that you wanna bring on in the early team. One is I think having a real domain understanding is really important.

[00:18:12] And so what I mean by that is. I think of data science, actually, a lot of times what a data scientist does is that it generalizes human intuition into data and algorithms. And then you subject those intuitions to a back test to see which of these intuitions are actually generalizable, but the core,  idea behind that statement is intuition, right?

[00:18:34] Like, so a data scientist has to have a really good understanding of the domain, so okay. To make it more concrete, you know, when I was at square, one of the best, uh, things I did there was I actually, um, mocked into issues of our operators and I caught up, uh, potential fraudsters. Right. And it was a really mind blowing experience cuz like, you know, you are seeing this whole GAT of people who, who are real merchants and pretend merchants and you know, some of those conversations are, uh, quite colorful.[00:19:00] 

[00:19:00] Um, in the case of open door, Uh, go in valley homes, right? And so you are sitting, uh, you're sitting alongside, um, folks in our pricing operations team. You're  going out to the field to see the houses. Um, you're sitting next to our homes team and figuring out, um, how to think about renovations, right?

[00:19:20] And you're on the hook for actually doing some number of valuations, uh, when you're onboarded as a data scientist here. And so these are some of the ways in which you can really develop domain understanding. And I think that is super important because you can't have an ivory tower, um, you know, in data science,

[00:19:37] Brett Berson: I I'm just so curious, uh, about the fraudster conversations. Are, are there are story or stories that come to mind in terms  of other than I'm sort some bizarre things that happened, but sort of maybe how, how that work translated back into what you were building.

[00:19:56] generally speaking, I I'd say, um, you can pick up [00:20:00] on some signals that are helpful at large and also some tools enable operators to do more efficient account review. And so for instance, um, in credit card fraud, uh, cart, not presence, meaning, you know, a payment that was entered via, you know, via manual entry.

[00:20:19] Those are, have a higher risk of fraud than cart presence, meaning the cart wasn't there and swiped for a transaction. And so when you see a lot of car, not presence for a, you know, for high dollar amounts for, um, uh, transactions that are supposed to be for travel at a later date or concerts for a later date, you start to question why, um, and you start to be able to piece together.

[00:20:43] You know, some of the signals are very relevant, like. Okay. Well, if you're a, uh, travel agencies that are booking, you know, multi thousand dollar travel, uh, bookings that are for a very much later date, all car not present, and you're seeing some amount of fraud, like you talk to these people and they have a very, [00:21:00] you know, thin story around how their business is constructed.

[00:21:03] You, you have greater fiction that, uh, you know, these are some of the signals that are promising or account review. Like, you know, how do you, uh, understand for an, uh, for an operator, the types of questions that they ask and how do you actually make those questions, uh, show up on the tool. So it's guided.

[00:21:22] And so a lot of those practice also apply to open door, right? Like what are some of the things that, uh, you look at from a data element standpoint, when you value home that might be gotchas, right? Like things that might have an outsized impact on the value of the home one way or another, how do you start to, uh, generate those elements in the UI, in the tools and capture that data so that it informs her algorithms as well.

[00:21:47] Brett Berson:  So something that you touched on a little bit, as we've been talking is, um, is the early team in the data science function. So I'm excited to hear more about what did that look like for open door? Maybe? Why did you structure it that [00:22:00] way and maybe what does that talent look like in the early days?

[00:22:06] And  does that change over time? Yeah.

[00:22:08] Ian Wong: yeah, that's a great question. And then absolutely changes over time. Um, but before talking about, you know, what, what changes I think, as we think about an early  data science team in a startup. Let's think about the context right in this early phase. Number one is your data science work band is really immature.

[00:22:26] Meaning your engineers are probably more focused on shipping the products and you might have some data engineers  as the case with open door, but they're probably gonna be focused on data ingestion, entity of resolution, right? Like for us at open door, real estate, data is messy. Uh, and the sources of data is very fragmented between recorder, offices, assessor offices, MLSs, and other types of, um, geospatial data sets.

[00:22:50] So our data engineers are just really focus on just pulling all that data set together. And so again, the data science work bench is immature, um, right. And that's number [00:23:00] one. Number two is the problem is not well defined. And in some cases the data assets aren't even defined and you have to go out to get the data or work with engineers to specify how the data will be collected or cleaned up.

[00:23:10] So your work isn't gonna be specked out. And in fact, your job is to spec out the problem. Your job is to spec out the objective function and that's not given to you. And the third is, um, the company's evolving very quickly, right? And so you have different teams of different types of data science problems.

[00:23:28] And so there's a, you know, it's an amorphous blob in the early days in some, in some ways. And so if you think about those, that context, the early startup data scientists, I think have to have a few really important attributes. Number one is I think first and more foremost, they have to be a self starter.

[00:23:48] Um, and I think that actually filters out a lot of data scientists, unfortunately, because I think in a larger company, there's just so much affordance and so much, um, infrastructure that you can rely, [00:24:00] rely on. And then the absence of all that affordance and all that infrastructure, it's hard for some of these individuals to be productive.

[00:24:06] So number one, just being a self starter, number two. And I think this is the part that gets a lot of data scientists who might come out of academia is that you have to be a sponge and you have to be deeply curious about the business. Um, you know, it's, it's easy to fall in love with the method, especially coming from a academic background, but what's more important in, in industries, falling in love with the customer and falling in love with the business problem.

[00:24:30] Um, and so generally I call these folks that we were trying to look for in the early days T-shaped individuals where there's an area of expertise, right? Like data, data, science, machine learning, statistics, what have you. But that top of their skill set is wide enough so that they can collaborate effectively with, um, other functions.

[00:24:48] And they're curious about the  other functions and they have enough of a skillset  to be able to unblock themselves.

[00:24:54] Brett Berson: So when you think about these kind of two  big characteristics, self starter [00:25:00] sponge, how did that map to where you looked for these people or what the ideal background was? I.

[00:25:07] Ian Wong:  generally speaking, I think a lot of great, um, early data scientists or data scientists are suitable for early stage startups. They either have had experiences in other early stage startups. Um, they might be early in their careers or maybe even in grad school, but have had shown a lot of independence and autonomy to try things on their own or to, you know, participate in cargo competitions.

[00:25:35] And in some cases it could be in larger companies, but again, you have to have a stronger filter for whether or not they are able to bootstrap themselves. Um, but as we think of thought about the early team  At open door, you know, I was able to pull from my network from, from square and from some of the, uh, other, uh, networks that we had in the founding team, like Eric's network from Trulia, [00:26:00] um, and JDS network from volunteers.

[00:26:01] So we're able to get folks that have really valuable early stage tech experience. And I think that was really, really helpful for us.

[00:26:11] Brett Berson: For the folks that you didn't work directly with, what was the interview or evaluation process and maybe how did that map back to kind of those two big characteristics and, and, you know, said differently if,  self starting is critical for a data scientist to have success in the early days at open door, what were you doing before you decided to hire them to get to conviction that  they would sort of spike in that area.

[00:26:37] Ian Wong: Yeah,  I spent quite a bit of time thinking through that, especially in the early days, because you know, in a larger company, you have a pretty much standardized set of questions that you ask. And these questions tend to be kinda like case studies or, you know, things that are not, to be honest, like directly relevant to the work that you're trying to do.

[00:26:53] And in the early startup phase, like you can't afford that. Like, you know, when you're bringing the person they're gonna work on the thing that needs [00:27:00] to be done. And so you all, like my philosophy was how do I mirror the interview process to mimic what they need to be doing on the job as closely as possible.

[00:27:10] So I know they'll be great at, you know, on day of one. And so the interview process was literal. Literally I took real estate data, um, from the many sources that I, uh, we have. For, uh, you know, for data lessons and purposes, I like actually spent some time randomizing, some of that data, Deon, uh, anonymizing, some of that data, but generally speaking, you're working with for all intensive purposes, the data set that we are working with.

[00:27:36] And then I ask you to build a model. I ask you to actually like build a model within two or three hours. And the idea is, again, that tracer bullet mentality, like, can you go end to end? Like, where are you getting stuck? And I'm, I'm sitting with you, like I'm spending hours and hours to these candidates.

[00:27:51] I'm sitting side by side to make sure that if there's something trivial, um, that they're stuck on, like I'm here to unblock cuz in real life, I am here to unblock you. [00:28:00] Right? So we're, we are gonna be doing a lot of pair programming together. And so I, I basically simulated as much as possible what the early stages of opener looked like, which was working with a lot of.

[00:28:11] Fragmented fragmented data sets having to boost strap your tool chain, meaning there isn't like, you know, this nice Google infrastructure. That's gonna allow you to, you know, ship models and, and, uh, using their, their software. Like you have to figure out how to use a lot open source. Um, and we just kind of worked through the problem together.

[00:28:29] So that's, that's how I made sure that the candidates were being assessed, um, in a way that mimics what they have to do, uh, in production. Yeah.

[00:28:41] Brett Berson: Can you think back to any hiring errors that you made in the early days where you had to ask someone to depart and was there a thread that tied any of those early mistakes together? The thing that you sort of got wrong in this evaluation process?

[00:28:55] Ian Wong: I mean, I, I think generally speaking [00:29:00] what's really important in the early days is again, the self starting, um, capability. Um, and that's actually in many cases, much more important than  existing domain expertise. And so what I mean by that is you might have folks that have many, many years of experience in a related field or  I saw this at square two, like, uh, payments or an open doors case real estate, and that experience is valuable, but there are other ways to come up to speed without experience very quickly.

[00:29:30] And I'd rather over index on your ability to unblock yourself and your ability to get, uh, up to speed very quickly. One, one phrase I love from the professor at Stanford was, um, a bit of slope mix up for a lot of intercept, and I think that's very true. Um, and so I think where we've made hiring mistakes, especially early on, was calling in quote, unquote, the specialists or the experience hires.

[00:29:58] Uh, a bit too early, [00:30:00] uh, and if they don't have, or they lost a skillset to be that self starter, I think it actually can create, um, quite a bit of problems.

[00:30:08] Brett Berson: As we've been talking about the, the early days and, and some of the recruiting ideas at, at open door, I'm interested just more broadly given the, the folks that were around the table with Keith and Eric JD and you, what, what have you learned or figured out as it relates to culture? Because I think that's, that's a, a big idea that powers all of your work.

[00:30:29] And I think you're all quite opinionated about it and maybe approach it a little bit differently than some other companies. And so what, what, what have you kind of figured out or learned about culture?

[00:30:40] Ian Wong: Certainly we had and certainly we had an opinion set of founders that's for sure. Between Keith, Eric, JD, and myself, that's for sure. And, you know, not all of our opinions are always the same. Um, but I, I will say one thing that I, I think we learned is, um, It's important to actually name the values, uh, because [00:31:00] they could be very implicit and sometimes where you have clashes and decisions, it could be a result of having misalignment and values and actually having a discussion around what the values mean is really important.

[00:31:11] So for instance, um, build openness is one of the, our, our earliest values and continues to be our value today. What does that mean? Right. And like, you know, uh, how do you actually practice it? It's easy to practice Val practice values when things are going well, right? Like when everything's up to enter, right?

[00:31:30] Like it's easy to share, like, Hey, here's how many contracts we're signing every, every week. Here's the number of hires we're making. Here's how, how much, how many dollars are flowing through the system? That's easy. Uh, valleys are tested when, well, shit like, you know, we haven't had a contract in a few weeks, you know, in the early days, I still remember. Having convinced great engineers to leave the apples and Googles and squares of the world to join open door. And like we had a really slow winter [00:32:00] and folks asked me one by one in our one on ones, Hey, what's happening with the business? And I think these are the moments where, um, build openness matters, right?

[00:32:11] Like being honest and about where the business is, what the opportunities are, but what the risks are. And that's how you can really enroll, um, you know, adults and really talented create creatives to be part of your journey. Um, so I think being intentional about culture, being able to specify what your values are, um, really, really matters and actually translate those values into rituals.

[00:32:35] Be it again, like, you know, the one-on-ones, but also like the all hands, the retros, um, you know, how we share our, our board decks, especially in early, early days, like all, all that are really important.

[00:32:48] Brett Berson:  can you walk through some of the early values and, and maybe why you all landed on them?

[00:32:56] Ian Wong: So some of our early values are still here with us today. [00:33:00] And, um, maybe there are two or three that I can highlight that's relevant here. So we talked about build openness that was with us since pretty much the beginning, the second is start and end with a customer. Um, and in fact, actually that's our first core value, which is to start and end with the customer.

[00:33:16] That's super important because I mean, ultimately both, you know, all the founders came together to start this company because we really believed in the impact of the work that we did for, for society. Like we, we genuinely believe that, like I personally really, truly honestly believe that, um, if we can do what.

[00:33:35] Promise to do an open door at scale, you will see more geographic mobility. You'll see people have more financial freedom slash opportunities. Um, and you'll see people be more embracing of risk. And that's really why I got so excited about the company and that I think it's a really important value to highlight because  Real estate is not an area where, you know, you transact every day, [00:34:00] right? The average time between transactions is typically seven years. Sometimes is five. Sometimes is 10, but on average is, is about seven. And the average age of, of a first time homeowner is about 32 nationally. Now these stats might have shifted over the last couple years, but it's roughly on that ballpark. And so that means that for. An, um, you know, for a new employee who shows up at open door, they might not have gone through a realistic experience themselves. They might not have bought a home or sold a home or even been to an open house. Um, and so how do you actually, uh, really help them appreciate the business domain, the, and the problems that we need to solve?

[00:34:40] I think that's a really critical business challenge it, and I really believe that, you know, employees that really understand the business problem and the customer pain points are the best employees because they're able to apply their creativity and the context of the problems are relevant to be solved.

[00:34:57] And so I think that's one of the key [00:35:00] values that we placed, uh, early days as number one, starting, starting, and then with customer, like we flew our entire early team out to Phoenix. Because  SF was our headquarters. We all went to Phoenix and all of us did customer studies. Like we talked to potential home shoppers.

[00:35:15] We talked to homeowners who were looking to sell a home. We got really deep  within a psychology of, you know, dozens and dozens of folks who were, you know, shopping or, or selling, uh, for  other homes that was critical. And we continuously did that from day one all the way, uh, to today.

[00:35:32] Ian Wong: of the real, real challenges of having an  SF headquarters while our products. Uh, was geared towards what I consider as mainstream America, meaning our products, uh, in the early days, uh, was geared towards Phoenix, Dallas, uh, Atlanta Vegas, like, you know, where the average price points of the houses are more like the average price points of houses nationally, unlike a San Francisco in Boston.

[00:35:57] So one, one [00:36:00] aspect that was tough for, uh, employees starting off in a San Francisco headquarters was having this appreciation for, for what we do and the impact that it has on our customer's lives. And so a lot of the focus that we had was around actually just setting our, our folks to the field. And so we were very intentional about, you know, to extent possible, um, having them, uh, travel to one of our, um, one of, one of our markets.

[00:36:26] And so in fact, we, uh, opened up a Sacramento market. We started operating there and one of the things we did was we actually had a bus every week. Um, once or Thursday morning, I forgot now, but that bus would pick people up from the SF headquarters and go to Sacramento and we would do a house tours.  and then we would shadow every single operational functions.

[00:36:47] And so we did that every week. And so that, you know, S an employee and open door, if you're an SF, there is no reason why you still can't understand how everything is done. Yeah. Early days[00:37:00] 

[00:37:00] Brett Berson: of closing out this, this opening section around, um, data science. The thing we didn't talk about is how the, the team was configured in the early days. Like what was the, when it was the first five or six people? Like what, what were the different skill sets on the team  and how was it organized?

[00:37:20] Ian Wong: kept it simple. Um,  so the whole engineering organization and data science organization, roughly had just three. Large teams. One was caught the product engineering team. So the team that was responsible for building all the external facing products and, and that team actually also built, ended up building a lot of internal tools.

[00:37:38] Then we had the data engineering team that was responsible for ingesting data, doing, uh, entity resolution, so that we have a clean data set to power our products, and also to enable our data scientists to do their research on a third was our data science team. So those are the three teams that we had initially.

[00:37:56] And then over time, obviously, then you have specializations and you kind of [00:38:00] split things up. So the data science team, you know, you had a predictive component to it, then you have a descriptive component to it. So you start to, uh, you know, split up according to those specializations all the way to now, you know, one of the things that I didn't fully appreciate until, um, later is the importance of specialists.

[00:38:15] You know, we talked about the importance of generalists early on. But it's also really important to figure out when in your journey you start to need specialists. And so over time we, you know, created more teams and more, uh, uh, spaces for folks with a lot of experience and who specialize in certain areas to be able to come in and be very productive.

[00:38:38] Brett Berson: Can you share some examples of, of sort of an indicator that you need to shift from this generalist orientation to a specialist orientation?

[00:38:48] Ian Wong: Yeah, I'd say there are, um, a few signals that come into play. Um, one is  when your, um, problem definition starts to stabilize [00:39:00] a little bit more. Um, and also when  as the CTO, and co-founder when I started talk to more folks from the different.

[00:39:08] Disciplines out there, you start to form a mapping of what disciplines are most transferable in their skillset, to the problems that  you need to solve. So as your prom definition, stabilizes, and as you have a better picture of the types of data science talent out there that is most suited to solve those problems, you might wanna start to tap into those specialists.

[00:39:28] So in our case for, um, at open door, we're trying to build an asset management system. We didn't really use those terms in the beginning. Um, but over time we realized, Hey, there's a lot of analogies to how, uh, hedge funds think about asset management. And now that skillset actually that financial thinking and inventory management mindset is very transferable.

[00:39:48] So actually qu are a great source of talent for us. right. As compared to, you know, if you are in a FinTech, you might wanna look to banks which are slightly [00:40:00] different, or if you're, uh, at Google, you might have, you know, a different need, a different set of, um, recommended systems or, or ads, machine learning needs.

[00:40:10] So again, understanding your pro domain a little bit better. And under, when, as those stabilize, having a clear talent mapping allows you to start to pull in the specialist. Cause at some point actually that's how you can accelerate your progress.

[00:40:26] Brett Berson: So shifting a little bit. And  you touched on some of these ideas, but you know  some of the things that I think makes open door so unique, and the story unique is, is this interplay between data science  and sort of real world operational components. And so I'd be interested.

[00:40:42] Maybe you could tell the story about kind of the interplay between those two functions and maybe what you think you got, right. That has allowed the business to thrive, particularly relative to so many  technology companies in the last 10 years. That that [00:41:00] ultimately just didn't work because  this specific interplay or figuring out how to get operating leverage over time was never realized.

[00:41:09] Ian Wong: hard, right? Like, uh, real world data science, uh, real world, uh, you know, whenever you're trying to build a business, that's actually operating the real world. You have to be able to think about operations and data science and engineering and products, et cetera, as all components of a system that has to work together.

[00:41:30] Um, and I think the things that we got right through a lot of trial and error, by the way, not something that we intuited from the beginning to be correct, um, is how to. Have effective collaboration between operations and data science. Um, that's one another is how do we think about scaling operations? Um, and number three is how do we think about, uh, creating competitive notes as those two, uh, aspects work together?

[00:41:58] So let me explain, [00:42:00] um, how do, how do we collaborate between operations and, and, um, and, and data science. It's hard because let me name some of the things that are hard about it in operations. Oftentimes you're measured on outputs, uh, in, in the matter of, on a time period of days in data science and engineering.

[00:42:20] Oftentimes you're measured in time periods of a quarter or two. And so the initiatives and the language and the work can look very different. And so there isn't really a common share framework for how to collaborate together and how once work feeds into another. Another challenge is that operational playbooks are easy to change.

[00:42:43] Um, relatively speaking, I mean, you have to train and you have to educate a lot of operators whenever you change a playbook, but generally speaking, it is easier to change an element in that playbook than it is to craft a very different products or a different set of tools. Um, [00:43:00] the lead times are different.

[00:43:02] And so what ends up happening is you might have diverging processes, uh, from the operation side as compared to the engineering side. Or sometimes because, um, you know, the case of data science, you can be really siloed very easily. You can start to really work on a small piece of the problem. That's not relevant to the end to end, um, use case.

[00:43:22] And so in these cases, one of the things I think we got right was, Hey, one of my best friends, you know, in the company has to be the COO or it has to be the chief customer officer in our case. Um, and the best friend for our VP of engineering who works in, uh, our, uh, operations and transactions platform, one of her best friends has to be her counterpart in operations as well.

[00:43:46] So first of all, there has to be real tight alignment at the organizational level. And then secondly, when it comes to the tactics, um, it's really actually important to have a process blueprint that all of us are aligned on. [00:44:00] I think actually this is a secret sauce slash like an element that a lot of people overlook.

[00:44:07] When you're talking about operational playbooks, there is an implicit and hopefully explicit process at play. And this process can be visualized in, you know, BPM like business process notation, or, you know, what have you like flow charts, whatever tool you use to visualize that process. It is very important to make that explicit and have that process be aligned between operations and engineering.

[00:44:29] Cause what ends ends up happening, especially in early days of a company is the operational playbook is changing by the minute. Uh, because you know,  there are customer exceptions or maybe you're trying to roll new products, it's faster to change operations. And so your engineers than data scientists are constantly building to a spec.

[00:44:45] That's like, you know, like three versions behind the latest version, so to speak. And so having those, uh, playbooks be crystallized and aligned on so that you know, that you're building towards the same future together 

[00:44:56] Brett Berson: You sort of started by talking about what makes this interplay really tricky, this [00:45:00] idea of, um, collaboration between ops and data science, um, difference in operating cadence, this sort of ops playbook idea. Maybe we could explore the other things that make it really tricky and the way that you approach solving it.

[00:45:16] Ian Wong: I think one of the, um, really nice things about operations, uh, or scaling with operations that could also be, um, harmful if done unchecked is, um, scaling it rapidly to the point where there's, uh, you know, the degree of centralization versus decentralization and. Uh, the repeatability of the process and our standardization of the process.

[00:45:44] What I mean by that is at open door. For instance, today, we are mostly centralized in our roles where, where possible we centralize. So we've centralized everything that doesn't need to touch the house in order to have faster scaling. So [00:46:00] even like how we, uh, assess the home today, we have self-service virtual assessment.

[00:46:05] That's done by a central team of, uh, customer  experience associates. Um, our pricing team for the most part is centralized so that they can really share their knowledge and have a centered process, but at various points in open doors life, especially in the early days, some of that was decentralized, meaning different geographies for instance, might have a slightly different set of, uh, operating procedures.

[00:46:29] And that's fine in some sense, because ultimately different geographies will have different. Requirements in some ways, but you wanna make sure those deviations from the center practice are controlled slash monitored because if not, then  it's really hard to scale that, and it's really hard to build tools and software to encode that those processes, you know,  I still remember  one of the moments that I learned a lot from was I remember like one of our teams who was responsible for is a specific tool, [00:47:00] was getting really frustrated because they  spent all this time aligning with their  operating partner and built this tool that everyone was very excited about.

[00:47:06] And there was no adoption. And so the team was very frustrated. So I remember actually flying out to one of our markets and just like seeing like how this tool being used in real life. And then I realized actually the in the field operators have a slightly different understanding of the process as compared to the central team.

[00:47:25] And so what had happening was the engineering team was trying to align with one set of operators, but actually  that process wasn't aligned with another set of operators. And you can imagine if you have end cities with end variations, this becomes an intractable problem. And so this is where centralization, where possible really being rigorous on S and playbooks, which we have now done at open door.

[00:47:48] Uh, again, through a lot of trial and error. Like those are some of the tough lessons that we've had to learn along the way.

[00:47:55] Brett Berson: And so how do you think about that sort of pendulum [00:48:00] between centralizing certain functions and decentralizing them over time? Like now that you have this sort of ability to look back in what worked and what didn't do you have, like a worldview of like in situation X, we should be focused on decentralizing it in situation.

[00:48:20] Y we should be focused on centralizing it or swinging the pendulum backwards and forwards in some sort of intentional way.

[00:48:30] Ian Wong: generally speaking centralization has the advantage of scalability, right? You can standardize processes. Uh, you can build software and algorithms to scale that out. You can really build a lot of efficiencies. If more of your team do things, uh, in a similar way at the same time, when you have a really uncertain environment that you're you're in and you wanna really, um, you might wanna really push accountability to the edges, um, so that they can have all the levers they [00:49:00] need to, to react to that circumstance.

[00:49:03] So it's not unlike actually in. Products engineering, where sometimes for new products, teams, product teams, you want that to almost operate autonomously. You want to operate, have enable them to operate as autonomously as possible. Like if you have a new product area that is fairly distinct from your core, I mean, you want that team to be able to be decentralized, so to speak.

[00:49:27] So there are values to both. I think there's a lot of value to running things in a  decentralized ways for the purposes of learning quickly. But then I think the art becomes okay. When do we think now this learnings  have been had. And actually the process has become more repeatable and now you're in kind of a scale mode.

[00:49:45] And then I think you really have to think hard about how fast can you centralize.

[00:49:50] Brett Berson: In, in this theme  of operations  and running and scaling and operationally intensive business. Are there other big ideas [00:50:00] that have powered the, the success of open door or maybe things you've changed your mind about or thorny problems that you figured out a unique solution that you think have enabled you to be so successful in this area?

[00:50:16] Ian Wong: So, I mean, it's funny cuz you know, I'm the CTO and my counterparts and operations are probably laughing at, you know, how naive I am still about operations and you know, my lack of understanding there. But, but what I will say is some of the things I've come to appreciate, especially as a technologist is, um, operations can be a real strategic differentiator.

[00:50:35] And in fact, um, some of these operational touchpoints are essential for the customer. So for instance, We're helping  homeowners sell their home. It's a single largest transaction that  most of them will do in their lives. And so having a human touch is really important that you sometimes just want someone to speak to it, to make sure that there's, that assurance on the other side of the phone.

[00:50:55] Um, and so we need to be able to support that and actually make that [00:51:00] a great experience for our customer. So that's not a defect that is a bonus, and  we need to make that a great, great thing for, for people, um, both, you know, for a customer and our internal operators. The other thing to really appreciate over time is the data that your operations team generate is proprietary first party data that accumulates and contributes to your data mode.

[00:51:26] And so that's actually oftentimes  some of the richest and most valuable data. That you have. So for instance, when we value homes,  we have our staff actually collect a lot of data above the house. Right. And the nice thing about having not just desk valuation is that we actually have folks in the field who go and inspect the home.

[00:51:47] And so for every home that we inspect, we have hundreds of data points about that house. And over time, as we've done  hundreds of thousands of these, we have like millions, if not tens of millions of data points about houses  in the United States. [00:52:00] And the extension of that is once you know, a thing about a home in a neighborhood, you actually know a bit about the characteristics of homes elsewhere in that neighborhood.

[00:52:10] And so that data mode starts to accumulate and compound over time. And that's all enabled because. Your operations team is talking to your data. Science team is talking to your engineering team, and you're trying to figure out how to build this feedback loop. Um, so I'd say, you know, really viewing operations as a strategic differentiator and how you can actually create that.

[00:52:33] Differentiation is something that I've come to appreciate over my time. 

[00:52:37] Ian Wong:  Um, some of the things that Eric and I, uh, and others in the early team we used to do is, uh, pre-mortems right? So, you know, the concept of post-mortem when an incident happened, uh, you retro and you figure out what, you know, the, you ask yourself the five whys and you have action items to prevent this thing from happening.

[00:52:54] Again. One of the things that we do did and still do [00:53:00] is the concept of premortem. So suppose we wanna launch an initiative and it doesn't work. Like why wouldn't it work? Like let's list all the reasons why this wouldn't work and let's actually work through the list of things that will prevent us from being successful.

[00:53:14] That's one, um, the other, which is a brutal thing, uh, we do, uh, is like, we play this game called whys. We tell ourselves. It's a great way to bust your ego and Buster, uh, Buster bubble that you live in. Like, what are all the things that you think like everyone want think is true. But actually we think there are actually lies that we tell ourselves, um, and it's not necessarily like real lies that they are just like fears, really?

[00:53:38] What are the fears that we have? So I remember some of the early days it was like, oh, you know, being able to sell your home, uh, nearly instantly is a product people love. Is that a lie? We tell ourselves is this product actually something that people want? I mean, obviously eight years later, you know, we can definitively say yes, if you give people certainty and, uh, and liquidity consumers crave it.[00:54:00] 

[00:54:00] But in the early days, you know, we let, just laid our  fears bear, and we kind of make sure that we were real with, uh, the problems that we had to solve. Um, and that, that kept us on our.

[00:54:12] Brett Berson:  Are there other rituals just when you zoom all the way out, you've been running different portions of open door for so long since day zero. Are there other rituals that you've relied on or weird things that you do as a company that you found tremendous value in?

[00:54:30] Brett Berson:  So to wrap up, I was hoping we could  talk about the role of the CTO and maybe what you figured out about that and, and coming into open door. This is the, the first time I think that you've started a company and been in the CTO seat. Um, and so  what's sort of the steepest learning curve over the past eight years?

[00:54:47] Or what are the things that you've had to work really hard to figure out or areas that you personally had to grow? Kind of going from maybe more of a, an engineer managing a few people. [00:55:00] um, to now sort of running the function at scale in, in the CTO role.

[00:55:08] Ian Wong: look, there are a lot of different challenges that the CTO faces and it changes at every stage of the company. Um, and so I think it is a bit meta up, but I've had to learn how to learn more quickly. um, I know that sounds weird, but like, cuz you know, from the get go that you're faced with like functional responsibilities as an engineering leader, like how do you ship the product or build the algorithm, grow the team.

[00:55:33] Like in some cases you have to where like the PM had or the analytics had, then they were like the co-founder and executive responsibilities. Right? What is the overall company strategy? Like how do you hire the senior team? What's the culture? Like how do you fundraise. And so there's never a dull moment.

[00:55:50] Like the, the challenges are constantly evolving. Um, and I think what I've learned over the eight last eight years is how I can [00:56:00] grow very quickly, uh, because the challenges do, uh, present themselves so differently and so quickly in succession, it's almost like once you solve a set of problems, you earn the right to solve the next set of problems and those just keep evolving.

[00:56:13] And so what I've found to be helpful is how to develop like a mentorship network, like getting a coach or getting advisors to the company, building kind of a network of peers from other companies, um, interviewing people, uh, you know, by, and just asking them how, how their job is done, especially in an area that you're trying to get up to speed on.

[00:56:32] I think learning to learn, I know it sounds a bit, meta has been a thing that, uh, I've realized is really important, um, and kind of allowed me to keep going for, for the last few years.

[00:56:45] Brett Berson: Are there other parts to that process of learning how to learn for you, or, you know, if a co-founding CTO reached out and said, Hey, we're 10 people or 20 people. And I want to [00:57:00] sort of adopt that idea and make sure that I'm growing faster than the business is. What's like the how to behind it.

[00:57:07] Ian Wong: Yeah. I mean, one of the things that I find really helpful as an exercise that you can, anyone can do, um, is once in a while, um, write down what your job wreck looks like. It sounds kind of weird, but like, um, if you were to hire your replacement, right, or if you were to say, Hey, here are the things that the company needs out of my role in the next 18 months.

[00:57:34] What does that look like? And I think  writing enforces clarity, and it's really important to be clear on what we want, um, going through that exercise of what is my job spec. What does a company need out of me over the next, next 18 months? I think that is a very clarifying exercise that I actually do on a regular basis.

[00:57:53] Um, I think that helps set, you know, set up how I think about my role and how I think about setting up my team. Yeah.[00:58:00] 

[00:58:02] Brett Berson: What does your job spec

[00:58:07] Ian Wong: So I'd say there are largely, um, three categories that it falls under strategy, uh, people and execution. So from a strategy standpoint, how do I make sure that open door, um, continues to be a technology company? Right? Like how do we, how does technology continue to play in outsize role in everything that we do?

[00:58:25] And that means collaborating with my exec team, having my team collaborate with our counterparts, to make sure that every roadmap is not just a business roadmap, but it's a business and technology roadmap. Another part is people like hiring is always challenging. Retaining is always challenging. Um, and so how do we make sure that we hiring the best.

[00:58:45] And growing our staff so that we have excellent, excellent engineers who are building, you know, the future of real estate. And the last thing is, um, more on the execution side. Obviously there are all these initiatives that need constant, you know, care and, and you need, you need to attend to it. [00:59:00] But  also the buildup of, of our various platforms that we need in the company across pricing, finance, transaction operations, consumer.

[00:59:09] So I'm really focused on making sure that our platform can meet the scale challenges and the innovation pace that we need. Um, you know, going forward

[00:59:17] Brett Berson: How did that wreck look maybe different than it did three or five years ago?

[00:59:23] Ian Wong: when we started the company, uh, my job mandate my job wreck was built the algo right. Or hire the first team. Um, so it was very specific. I mean, the general categories of like strategy people, execution. That's probably durable, but what falls underneath that? And the time horizon that you're expected to achieve these results are very different early on.

[00:59:44] Right. Cause in early days, like you're trying to get to an outcome in like three months, maybe, you know, maybe six months here, you know, as the company matures. Yes. There are gonna be urgent fires if to put out, but  I spent a lot more of my time thinking about, okay, [01:00:00] what about the next 12 months?

[01:00:01] What about the next 18 months? Like, do we have the right team structure, leadership structure? How how's our platform looking? So kind of your vantage point changes, um, as a team, um, you know, as the company matures.

[01:00:13] Brett Berson: When you think about  an evolution of the role as a CTO, um, how did you approach kind of the interesting question of, do you wanna focus on, um, improving your weaknesses versus just doubling down on your strengths?

[01:00:31] Ian Wong: It's it's an interesting question. Cause I don't know how other execs and other folks would answer this or other founders, but for me it's always a bit of both meaning. Um, you wanna make sure I wanna make sure that what I think of as my strengths, uh, like my intuition around how to solve technical problems, um, my intuition around how to, uh, construct a team, I wanna make sure I continue to be really, to [01:01:00] be fearless in, in being able to practice those, uh, and to express those skills.

[01:01:04] On the other hand, I know I have a lot of weaknesses, but it's not necessarily trying to, I mean, ideally I can turn those weaknesses into strengths, but I have enough humility to know after so many years that it's hard. Like there, there are things that I just am not good at professionally and, and I, I wanna get better at it, but I it's gonna be, it's gonna take a long time.

[01:01:24] So in those cases, almost like it's how I, how do I shock absorb some of the weaknesses I have, like some of the rough edges I have either through ideally self-improvement or in my case, you know, how do I construct a team around me that allows, you know, a focus on those areas? And so for instance, um, you know, I tend to do better being focused on one or two prompts at, at, at one.

[01:01:45] So I have a hard time, like multitasking. I have a hard time actually, you know, thinking through, um, like say some of our, our, our processes around like, um, the HR P uh, process around like promotion calibration. Like these [01:02:00] are really important things, but I, my mind naturally drifts to like, how do we improve the platform?

[01:02:04] Like, how do we like move to the SOA? And so how do I then construct a team IE, like, you know, who's the chief people officer that we need to recruit. How do I think about the VP bench I have and what, what the skills that they bring to the table? Like how do I make sure that together as a leadership team, that we have all the skills necessary to run a really healthy organization.

[01:02:25] Brett Berson: Awesome. Well, thank you so much for spending the time with

[01:02:28] Ian Wong: Thanks so much, Brett, this was a lot of fun.