Does tech debt matter if the user can’t detect the “smelly code?” In short, yes. The same can be asked to the fashion designer with a poorly sewn dress — technically the garment functions, but with use, the closer it gets to falling apart.
When engineers write code, part of the process includes cleaning up mistakes they’ve made along the way to prevent racking up technical debt. But as deadlines approach, sometimes it's easier or faster to continue programming without fixing those errors. The code itself is still usable even if it isn’t optimal.
Every engineer will face tech debt, but luckily, there are many methods to manage it. For comparisons ranging from foul eggs to car loans, read on to understand how two local engineers define and manage technical debt.
Going to the dentist is scary enough without having to deal with payment plans and insurance costs. Lead Software Engineer Lukus Reindl shares how he works with technical debt so that OrthoFi’s users are always smiling.
What is technical debt and how does your team define it?
At OrthoFi, we believe that the state of your architecture is akin to your physical health and fitness. That is, having a plan to account for tech debt would, in essence, be the equivalent to eating your veggies and exercising.
To us, technical debt can be defined and categorized through different reasons: scoping considerations as it relates to market delivery, the need to keep your tech stack current as new technologies emerge, and accounting for the scalability of your product as your business needs change and/or your client base grows over time.
No matter the reason, it is important to our team that we have plans to address tech debt so that we can support our growing business needs and, ultimately, delight our customers.
When technical debt does occur, what process does your team use to measure and manage it?
Using the Spotify model, we logically divide our ownership of an architecture area. As part of the team’s accountability areas, there are defined engineering metrics that we use to monitor and measure tech debt.
The three metrics we focus on include user story cycle time, human time spent on release activities, and a ratio of new feature development vs. maintenance on the product. We balance throughput with measures of quality and stability, including metrics around test automation coverage and webpage performance. Finally, we continually measure infrastructure cost and usage projections regularly, given we are a cloud-hosted infrastructure (AWS).
As guidance to the teams, we believe that metrics should not be used for judgments, but are rather used as a means to identify where we should be making proactive investments.
Teams are additionally encouraged to manage their team-level tech debt roadmap.”
What proactive measures does your team take in minimizing technical debt?
This is typically done by leveraging our cross-functional architecture “chapters” to organize our organizational technical roadmap into engineering OKRs, used to align efforts on a quarterly basis. Each of our Agile teams carve out approximately 20 percent of their team capacity to work against these items.
Teams are additionally encouraged to manage their team-level tech debt roadmap. If the teams have additional capacity left over within their 20 percent allocation, they are encouraged to execute against this roadmap. They can also argue that their team roadmap initiatives should take priority over engineering OKRs, which is a welcomed debate as part of quarterly alignment.
Finally, not all tech debt should be considered part of our 20 percent allocation. Items such as scoping decisions should continually be vetted and collaborated upon between product, engineering, and our business partners, using transparency to help guide good technical business decisions.
As the saying goes: if it ain’t broke don’t fix it. When it comes to the residential mortgage industry, however, Maxwell knows there’s a lot to fix. Vice President of Engineering Lee Gonzales shares how he manages technical debt to keep the business growing.
What is technical debt and how does your team define it?
Technical debt is any deficiency in a system that makes it harder to change, improve or maintain. Concretely, technical debt comes in at least three main forms.
Technical debt does not always start out as technical debt. Often, well-written code poorly maintained can turn into technical debt. Or code can turn into debt if it once served a purpose but is now no longer used because the business shifted or technical needs changed.
Technical debt can also take the form of a wise investment to achieve a business outcome, but much like taking out a loan to fix your car so you can still drive to work, it is best used only in a pinch or when there is an exogenous need, like an external deadline. Generally speaking, the lean thinkers have this right: go slow to go fast.
Technical debt can also be due to omission. Most often, this takes the form of not writing tests. What we usually see is that our test code coverage is too low. Or that our integration tests are faulty, flaky or incomplete. But like flossing, you just have to do it. Other forms of missing code can take the form of missing error handling, logging to enable observability, or metrics to properly understand system performance.
Technical debt can be caused by writing too much code or the wrong code. At Maxwell, we sometimes see this a lot when we’re building systems. The general guideline here is if you see something twice, then think about an abstraction. Then, if you write the same code a third time, refactor and optimize. You can get this same effect by continuously adopting the latest and greatest libraries (see Javascript) or newfangled practices. Change your software patterns often and the same effect occurs.
Lastly, when measuring technical debt, you should be aware that there is a dark form of technical debt.”
When technical debt does occur, what process does your team use to measure and manage it?
Much like measuring productivity, it can be very difficult to properly measure technical debt writ large in your systems. In other instances, technical debt is signaled via more and more bugs emanating from an area of your systems. Or maybe like eggs that have gone bad, there is a particularly foul smell you start to notice in parts of your codebase — if that occurs, schedule time to have a senior engineer investigate and make a recommendation on what to do in order to prudently handle it. Since it’s hard to judge the positive or negative long-term effects of technical debt, pay it down. The best course of action is to slowly and incrementally pay it down based on the hot spots in the system and/or areas where frequent updates are required. However, always remember that infrequently changed code which looks suspicious, but never breaks, can be just fine.
Lastly, when measuring technical debt, you should be aware that there is a dark form of technical debt. Senior engineers whisper about it in hushed tones, tiger teams get formed around it, and businesses fail because of it. It is the worst and most mind-bending form of technical debt. A bit like dark matter or a coal fire underground, this is technical risk below the surface of clear observation. This form of technical debt arises from unforeseen dependencies, creeping system complexity, or the violation of deep system constraints due to basic, system-wide assumptions. Examples include assumptions like, “We will never have a million customers so an 8 KB customer cache in memory is fine.” This type of landmine usually remains invisible until you have a snafu. This is what you want to avoid.
What proactive measures does your team take in minimizing technical debt?
Leave code better than you found it and steadily pay your debt down by allocating a steady and consistent budget to its eradication or management. Scheduling 20 percent of a team’s time is often a good guideline. Good situational awareness also helps, as can hiring experienced senior people who can spot the patterns of technical debt, or sniff out the scary smells of dark debt.
