In the early 90s, celebrated computer programmer and wiki inventor Ward Cunningham first synthesized the concepts of complex technical development and debt. The coined term gives a name to the sinking feeling that haunts dev teams the world round. He warns of the incremental debt that engineering organizations accrue in each line of effective — but imperfect — code that’s shipped. If it’s paid back promptly with a rewrite, all’s well. But if not, there’s interest on that debt that compounds and collapses on top of the engineering team that owns it.
Enter One Medical Group CTO Kimber Lockhart, the engineering equivalent of your savvy financial planner. Lockhart subscribes to a philosophy — inspired by woodworking — that helps startups judiciously control the creation and accumulation of technical debt. Before her current post, she rose through the ranks quickly over her four-year tenure at Box, where she started as a software engineer and grew to lead teams throughout the application stack. She joined Box after it acquired cloud storage and document sharing startup Increo, which she co-founded and led.
In this exclusive interview, Lockhart deconstructs technical debt and offers woodworking techniques that will better help you manage it. Like other shrewd technical minds on tradeoffs, she shares how to astutely approach shortcuts, especially in engineering environments where the need for both speed and integrity can be paramount.
Technical debt is not the scarlet letter. It happens to the best of teams. I’d argue it’s actually irresponsible for a startup not to have any technical debt.
How To Handle Technical Debt Before it Becomes Bad Code
The stigma around debt — technical or otherwise — is that it’s a sign of weakness or leads to overreliance, when in fact it merely represents a trade-off, giving more time or resources at a cost. “Even the highest-performing teams incur technical debt. The difference is that they do so as a controlled decision to take a shortcut. Shortcuts are typically bad, but sometimes can enable thoughtful choices in a growing business. When too many shortcuts are used to save time and technical debt compounds, bad code can result. The key is to keenly understand the tradeoff of shortcuts throughout the growth of a business,” says Lockhart.
For example, if your goal is to achieve product-market fit for a business, your priority is to iterate quickly to figure out if you’re building the right product for the right audience at a pace that’s viable for the business. "In this case, assuming technical debt by taking shortcuts may be worth it, even if it introduces bad code," says Lockhart. "This is not a blank check for bad code, but there is a nuanced use for it in order to tally wins in the short term to ensure the long-term. That’s the double-edge of technical debt.”
In short, technical debt isn’t inherently bad, but how it’s managed dictates the positive or negative impact on a business. Here’s Lockhart’s tips on how to use technical debt prudently:
Mix in master carpenters. “It’s common for young founders — and actually all team leaders — to hire too many engineers at the early end of the experience spectrum,” says Lockhart. “One easy way to prevent technical debt from generating bad code is to anchor green technical teams with veterans, who can call on past experiences to more adeptly navigate difficult decisions.”
At both One Medical and Box, Lockhart learned the value of leaning on seasoned engineers who had experienced enough technical and business outcomes to make good, reasoned judgments around accumulating and resolving tech debt. “They need to have seen something that they’ve personally built move the business forward, and they need to live to regret a shortcut they have taken. This is a time-intensive cycle, but there’s no alternative to personal experience.
So, founders, hire beyond your social circle. Look for engineers who have stayed at companies long enough to experience the long term impact of their decisions. Young devs, loop in a few pairs of experienced eyes. Vets, take some green engineers under your wing. “Having more perspectives is the key to leveraging the experience that people have in accumulating and addressing technical debt,” says Lockhart.
Measure twice, cut once. Begin code review before anyone codes. The best technical teams envision — not just review — code together. “It goes without saying that some form of code review is essential in any type of engineering organization,” Lockhart says. “Early on, pair development or sitting next to each other reading through code will work. As the team grows, it might be important to get code reviews more formally from different individuals or teams.”
“For many companies, process evolves so that this code review is the only time engineers get feedback on their code and iterate to make it better," says Lockhart. "Unfortunately, finding a problem after the fact forces the tough decision between taking the time to rewrite and living with bad code.”
Front-loading more discussion doesn’t automatically mean running an arduous planning process. “Early on, get everyone together around a white board, take a photo of it and share. It can be that casual,” says Lockhart. “The important part is letting multiple people weigh in on the discussion and writing down the result. What you want isn’t a long, formal design document, but rather to mine the wisdom of the group to get to the most pragmatic action.”
Collect and codify your standards. At first, carpentry know-how was largely passed down person-to-person. As the trade specialized and creations became more complex, woodworkers produced guides and pattern books. With his treatise De Architectura, Roman architect and author Marcus Vitruvius Pollio produced one of the most important, original sources of planning and design, standardizing how aqueducts to small measuring devices are built.
“It’s so simple, and so many teams don’t do it. There are multiple valid ways to do just about anything in programming, and the worst choice is to be inconsistent about your choice,” Lockhart says. “Better yet, encode your standards into a linter or other static analyzer. Start with an almost empty set of rules, and then slowly add as you agree upon standards for your code. Yes, it will take some heads-down time to get your code conformant with each new rule, but the consistency will go a long way to prevent bad code.”
Scrap the shortcuts that don’t save time. Woodworkers know when they can use shortcuts and when they just won’t do the trick. To refinish wood, the existing finish must be fully stripped from the wood — no wipe-on polyurethane will suffice. Taking the shortcut of not removing the existing finish ultimately takes just as long as doing it the right way, especially once the new finish is layered on the old one, dries and you have to fix the underlying flaws.
Similarly, Lockhart has found that shortcuts can be an illusion — often it takes the same amount of time to write clean code as it would to produce code that introduces technical debt. “The problem is, bad code often feels faster in the same way hurrying feels faster,” says Lockhart, “little time is wasted planning and the code itself is written more frantically.”
“The choice of ineffective shortcuts is typically not due to the task at hand, but the circumstances around the task. If your team is inexperienced or under a lot of unnecessary pressure, you’ll often get people picking less optimal ways of building something that doesn’t save time, energy or resources,” she says. “One of my first projects at Box was to integrate the document preview technology from my first company, Increo, into the Box product. I put a lot of pressure on myself to finish the integration quickly, and re-used some of the code patterns in place instead of thinking through the problem with fresh eyes. While my approach certainly felt like a shortcut, in hindsight a much simpler approach would have resulted in a more intuitive implementation that would have actually been faster to complete.”
“Leaders can counteract this tendency by hiring for experience, implementing early design and code review and avoiding unnecessary time pressure,” advises Lockhart.
Build modularly. For furniture that’s quickly built, taken apart and rebuilt, carpenters use knock-down fasteners, a type of joint that allows for fast dismantlement and assembly of modular parts. They facilitate flexibility and easy completion, even for a novice. Similarly, modular code is flexible, and tricky changes are easier to implement within modules than across a large code base.
For engineering teams, half-completed framework shifts are one of the most common causes of confusing technical debt. With today’s proliferation of front end web frameworks and rapid versioning of open source tools, the temptation to use the latest and greatest is high. Before switching, Lockhart recommends considering: “Is the slight increase in functionality that you get from the new framework worth the cost of rewriting your code?”
“You can make that equation look better by building your code as modularly as possible,” says Lockhart, “splitting those modules on break lines between features and functionalities. Then, the cost of applying a new framework is only the cost of rewriting a portion of the code.”
“At One Medical, we have a central API and data store, but different applications that surface that data to different audiences. Each of those small applications can run a slightly different version of our core frameworks. That way, you don’t have to make a change across your whole organization when you want to make a change for just one audience.”
What To Do When You’re Faced With Bad Code
Technical debt should be a controlled decision to take a shortcut. When too many uncoordinated shortcuts occur and technical debt is “overdrawn,” bad code happens. Here’s how to prioritize, label and navigate bad code resulting from sustained technical debt.
Create a rating system for bad code. “Many engineering teams lament the failure of their organization to adequately address bad code resulting from technical debt, but they can’t get their footing when asked how to resolve it,” Lockhart observes. “Engineering teams owe their organization careful prioritization, just like anyone else making requests.”
With the team at Box, Lockhart devised and used a system to rate and prioritize this type of bad code. To get a rating, the team would look at three different factors:
- Security. Are there security concerns associated with this bad code?
- Prevalence. How widespread is the bad code? How intertwined is it with different parts of the code base?
- Frequency. How often do you come into contact with that bad code? How often does it run when users interact with the product?
“After rating those three factors, a rough ranking of projects emerges,” says Lockhart. But no rating system is foolproof without a human scan to make sure the priorities make sense. An automated scorecard can’t catch everything.” Carefully prioritizing projects to resolve bad code is like paying off high interest credit cards first — it stops debt from compounding.
Technical debt is like poison oak. It’s naturally prevalent, but need not be ripped out to be avoided. A warning sign goes a long way.
Encapsulate and label your bad code. Engineering teams may feel pressure to either push ahead with bad code or immediately find and rip it out. Yet, there’s a stage in-between that is useful, less aggressive and not as time-consuming. “If there’s part of your code that you know is bad and don’t want people to repeat it, label it. With that act, you’re explicitly indicating which code should no longer be followed, even if it’s currently necessary for functionality,” says Lockhart. “It can be as simple as deprecating a function, or inserting a comment that indicates code that is no longer in the modern style of the your development team. It’ll go a long way toward keeping bad code from propagating.”
“The next step is to contain bad code so it doesn’t spread,” says Lockhart. “A simple example might be to isolate the bad code on an object and extend that object with neat, clean new functionality.”
“You will find something more in woods than in books. Trees and stones will teach you that which you can never learn from masters.” - St. Bernard de Clairvaux
Construct tools to counter bad code. To handle and nullify negative technical debt, Lockhart draws from two tools and techniques in woodworking: the shim and the jig. The shim is a thin, tapered wedge that fill small gaps between objects, while a jig is an auxiliary tool that controls the position or movement of another tool. Here’s how Lockhart applies these instruments to software development:
The Shim
In woodworking, the purpose of the shim is better alignment. “The shim is often a small piece of wood that’s inserted when two parts don’t fit together how they should. Sometimes it’s left in the final product to ensure better structure, but it also can be removed after it serves its role,” says Lockhart. “Shimming code is the notion of putting in some code to make two parts work together that might not otherwise. So, if a part of your code doesn’t expose the right interface, you can add a layer that makes it do what you want it to do.”
At One Medical, Lockhart and her team sought to extend the capabilities of its messaging service. “We planned to build this module that made messaging easy between doctors and patients. The problem was that we originally built the code to send a different type of message,” says Lockhart. “We feared we’d need to start from scratch, rewriting the whole system to accept the new type of message. Instead, we wrote and inserted a shim so that the old system could accept the new kind of message. We avoided spending months and months rewriting the old code.”
A common example of a shim is a software development kit (SDK), which developers might produce to make it easier to use an API from a language that the customer uses. “It’s a shim because it’s a thin layer of code that eliminates the disconnect,” says Lockhart.
“My favorite use case of shimming code is API versioning. It’s common for APIs to change as a company grows and iterates. To keep old integrations working, companies often choose to version their API. Inevitably, v1 of the API implementation has a lot of technical debt, and the code is bad. The opportunity here is to build v2 of the API, and then use a shim to the v1 interface to the implementation of v2, removing old code and allowing the APIs to work in unison.
Lockhart recognizes that most engineers may not think of this technique as a shim. “Good engineers do this all the time. But by giving this tool a name, we’ve noticed that we think of it as more of an acceptable option,” she says. “Instead of rewriting it completely, we can now say internally: ‘Hey, are you sure we can’t do that with a shim?’ It has recognition and people more frequently acknowledge it as a tool in their toolset.”
Lockhart offers a cautionary note on shims, “like any shortcut, overreliance on shims leads to technical debt and bad code. Make sure you’re using a shim to get it the last mile, not to do heavy lifting or major patchwork.”
The idea that you need to resolve technical debt immediately is not a useful mindset at all stages of a company.
The Jig
“More complex than a shim, a jig is a tool built by woodworkers that facilitates — but does not go into — the final product. For example, it could be a device that guides a saw to achieve a complex type of cut,” says Lockhart. “Jigs generally aren’t quick to produce -- there’s as much craftsmanship that goes into the jig as the final product. By avoiding a shortcut on a jig, you end up with a better final product. I once hated the concept of working so hard on anything that wasn’t a final product. But the investment in a jig can be entirely worth it.”
Applied to technology, a jig is any code with the primary purpose of enabling better code, not being shipped into production and used itself. Jigs are especially useful to provide support in the process of cleaning up bad code.
“The clearest example of a jig in this sense is test automation. Test code is written that exercises the application to ensure it is working as expected. The test code isn’t part of the final product, but it makes it better. Testing is the most important jig for avoiding bad code, every new feature should be tested and every bug fix should include a new test to prevent the issue. Even writing test code you intend to throw away is valuable -- sometimes the only way to safely modify old code is to write tests for it in place, rewrite the code, and then remove the preliminary tests in favor of more reliable tests on the new code.”
Jigs exist outside of testing as well. “Say I’m writing an API to send someone a text message via a web application. To thoroughly test that API, I may decide to produce a very simple client to exercise the API I’ve written. Ultimately I’m going to throw away that code -- It only exists so I can build a better API."
A component library, or living style guide, is another example of a jig, one that helps test and maintain front-end components. “When writing client-facing code, it’s a good practice to program with components rather than do it from scratch. Instead of building a button over and over again, the developer builds one button and re-uses it everywhere. This is an important part of how modern developers think about writing user interface code,” says Lockhart. “Twitter’s Bootstrap library is a great example. You can see each component, so if someone tries to modify a component, it’s easy to see unintended side effects by going to the library page, a page your users never see. The benefit is that you can isolate components, make changes to them and assess the impact. These are expensive jigs to build, but by investing that time you save hours worth of bug fixes and faulty deployments to users.”
On Choosing Shims and Jigs
Like most craftsmen, developers also have their favorite tools. “When companies build less and start iterating more to hit product-market fit, it’s smart to double-down on shims. With them, you move fast without creating a ton of technical debt,” says Lockhart. “Jigs are harder early on -- it’s hard to think about building code that you’ll throw away. But it’s a powerful concept if you’re aiming to create something of high quality.
Almost every element of your product will need to be revisited at some point in the future. Being unable to do that due to technical debt can be paralyzing.
Technical debt can be both a signal of and a strain on fast-growing companies. The goal is to monitor and manage technical debt, with tactics such as labeling it, creating a rating system and building code in a modular way. Employ woodworking techniques such as shims and jigs not only to mitigate technical debt but also to use it to help your engineering organization move faster, smarter and more collaboratively. Lastly, as you groom your engineering team, pepper it with veterans who have seen the full cycle of technical debt, so you can reward its collective ability to take a nuanced approach to making thoughtful tradeoffs.
Here’s a summary rundown of Lockhart’s observations on technical debt to take forward:
- “Technical debt affects your whole development process. Period. It can be used to make thoughtful choices in a business, or be a destructive force if neglected.”
- “Hire seasoned engineers who have some tolerance for technical debt and an earned intuition when it comes to trade-offs. Seek developers who think in pros and cons, not absolutes.”
- “If code is business-critical or going to impact customers, address technical debt immediately. But if the debt is not causing any imminent problems, monitor and let it go.”
- “Fixing technical debt doesn’t magically make all code easier to write. A double-edged sword, it can be both a lever for speedy deployments and fertile ground for bad code. ”
- “Channel your inner woodworker. Shims and jigs can help transform technical debt from a threat to your technical team to a tool that helps build and bolster your code base.”