The Agile Architect

Building New Software? Beware The Tail: It Could Destroy You!

Our Agile Architect muses on the total cost of ownership of software and the ongoing cost, after initial development, of paying back technical debt due to bugs, low-quality software, and lack of automation.

• By Mark J. Balbes, Ph.D.
• July 12, 2022

This is a story of Death by 1000 Cuts. This is a story of your system. You’ve been with it from the beginning. You are being asked to add a new feature. But there’s no time. You have to fix 25 bugs, run the 100-page manual regression tests procedure, prep for the bug fix release, update for a requirement, do the umpteen daily tasks that keep the system going, and write estimates for next year’s feature list.  Then you need to attend meetings to triage bugs, discuss why you are behind in implementation, and what you and the company are going to do about it. There’s no time to implement new functionality and you are already working long hours. How did you get here? At every step you made the right decision but now you are drowning in the minutia of keeping the system running and there’s no time to do anything innovative. What could you have done differently that would have resulted in a better outcome?

The Technical Debt Fallacy

In my career, I’ve seen how a lot of companies build applications. Everyone believes they have logical reasons for working the way they do. Some truly believe they are building software the right way. Some feel trapped, doing the right thing for their local situation but knowing there’s something better if only they could get to it.

I’d love to say that high-quality software is always the right answer, but I’ve also seen good products fail. Taking too long to get to market, they are brought down either by the realities of cash flow issues or a competitor beating them to the prize.

There is a common fallacy among agile developers that technical debt is always bad. In fact, it can be an important tool to achieving a goal. Just like purchasing a house may require a family to go into financial debt, sometimes successfully getting a product to market means taking on technical debt.

In fact, I only consider something truly technical debt if there was a conscious business decision to make the tradeoff between quality and cost or quality and time-to-market. If the development team is skimping on quality practices, it’s not technical debt. It’s just crappy code.

It's Not Just About Technical Debt

The real issue at hand is not technical debt, despite all the space I’ve devoted to it above. Technical debt may be the root of the evil, but it manifests in terms of total cost of ownership of the software.

Think about the cost of implementing the initial product/feature/release. The cost of an initial build may be higher if your team builds a quality product. The cost for sustainment (i.e., “the tail”), however, is less. Over time, as the number of features and applications grows, those tails grow to consume ever-more of the IT budget. With quality built into the product, the tail is minimal and sustainable, and the IT budget can mostly go to new development.

Building The Maintenance Tail One Bad Decision at a Time

If you’ve been reading The Agile Architect over the last 10 years, you probably know what I mean by building quality into the product. It means investing in an automated deployment pipeline. It means automated tests running in that pipeline supported by all the necessary infrastructure. It means following quality coding practices, including low or zero bug tolerance, ruthless refactoring, test-driven development, and pair or mob programming. It takes a dedication and focus on all these things.

Building the tail happens the opposite way, with a lack of discipline and focus. Here are some of the typical things I’ve heard people say as they build an impossible-to-maintain system.

It’s just a few lines of code
Spoken to justify pasting yet another duplicate line of code.

I can fix this on the production server
Spoken as an expedient to fix a production issue. Now the production system is out of sync with all the others.

I don’t need tests
Often heard with, “This change is so simple we don’t need to test it.”
May stem from the belief that manual happy path testing is sufficient. The users will do most of the testing for them. Accompanied by an arrogant confidence that they can fix the bugs quickly when users complain.

We’re going to need this
Justification for building an overly-complicated system with code for unused features that still has to be maintained.

Manual deployment works fine for us
Spoken by those who don’t have to do a manual midnight deployment and then scramble when things don’t go right.

Our users would rather have more features
Spoken by some to justify always building features rather than fixing bugs or paying down technical debt.

Final Thoughts

The cost of a large maintenance tail not only consumes an over-size quantity of the IT budget, it also leads to early retirement of the system in favor of something more manageable. This raises the total cost of ownership even higher, because you don’t get as long a lifetime from the product.

Finding the right balance between true technical debt and perfect quality is a difficult task. It should be done with a specific outcome in mind and with an understanding of the price to be paid later to get important gains now. Poor decisions disguised with an “it doesn’t matter” arrogance should not be tolerated or allowed to be passed off as technical debt.