Technical debt - having unfinished tasks in an IT project - is a significant problem for many organizations and can contribute to major development issues across a company. Developing strategies to prevent and deal with issues of technical debt can be incredibly difficult. To help you, we recently got together with Russ Miller, CTO for SunView Software to talk about the best ways to deal with technical debt.
Q. What are some of the primary causes of technical debt?
A. Technical debt creeps in any time short-term tradeoffs are made to address business needs without an appropriate investment in the infrastructure that supports that process or system. Thus far, technical debt has primarily been talked about with regards to application development, but technical debt is just as prevalent, and often just as costly throughout IT.
Here's one simple, concrete example just to make the point - technical debt is built up when a lack of care is taken in marking and organizing network cables; it is easy to mark them and organize them as they are added, but much more work to trace issues while they are disorganized. The longer you wait to organize cables, the more work and greater the impact once they look like a bowl full of spaghetti. A form of interest is paid whenever there is a problem to troubleshoot that requires knowing which cable leads where.
For IT application development, a good example is integrating code that is not covered by automated testing. Your organization pays "interest" along the way in terms of additional manual testing and longer cycle times to make up for the lack of automated tests; moreover, eventually you will likely need the tests and have to write them anyway, and since it is more efficient to write the tests alongside the initial development, those missing tests represent debt that must be paid back, eventually. The sooner it is paid back, the less total cost associated with not creating those tests.
The more serious debt comes from deploying a device or system with known issues in it that cause the solution to cost additional ongoing effort to use or service. An organization typically runs with that underperforming, under-capacity system for some time while looking for an opportunity to either improve it or deploy a longer-term solution. The overhead endured as a result of not addressing issues upfront is a form of interest, and the deferred cost of actually addressing the shortcoming is a form of debt.
Just as in the real world, debt is OK as long as it is used as leverage with reasonable odds of a return for the risk. It is also OK in IT to, likewise, amass debt in a calculated way, where it provides leverage and is likely to pay off. But all too often, that ROI analysis is not made, or like in the cable example, it is not OK if it creeps in due to plain old lack of process or discipline.
Q. What can development teams do to prevent technical debt from escalating into a major problem?
A. The first step is to be able to recognize where and when you are acquiring the debt, where that debt is "costing you compound interest," and start a process improvement or refactoring effort to reduce additional debt.
But how do you recognize the sources?
Places to find technical debt
1) A known error database - A database that catalogs areas where you have not gotten to the bottom of problems that are eating away at people's time (paying interest). This system makes it easier to identify which problems have the highest ongoing incident rate.
2) Recurring events - Incidents that frequently come up in the system log of key devices and applications often represent debt.
3) Compliance issues - Regulatory problems that are not directly handled represent debt and leave a window open for security or privacy breaches to meet a schedule. If you have been audited, look at the audit exceptions for possible sources of debt.
4) Change requests - Randomly pick change requests from the prior year. Looking with hindsight, which of those changes led to debt that you are now paying? How were the decisions reached?
5) Current changes - Look at changes or projects that are in the pipeline. Which of these represent re-work that would not be there if you had considered the longer-term costs?
6) Alternative locations - If these debt locations don't satisfy you, get up and manage by walking around at a random time each day. On average, what is the staff working on that you are surprised to find them doing? In all likelihood, these unexpected projects are likely being performed because of debt.
With the sources identified and prioritized, change requests need to be created and treated with equal importance to the requests for new features or systems that are in the backlog. There will never be time to stop everything and address all sources of debt. Instead, what is critical is to incrementally address the sources, bit by bit, while doing what is possible to not acquire new debt.
Q. How does technical debt eventually end up adversely impacting individual projects and the IT department as a whole?
A. Individual projects are harder to complete if those tasked to work on them are still paying off debt from prior projects. The schedule will likely assume they can focus on the new project, but the reality is they are pulled back into prior efforts when the debt of those prior efforts is getting in the way. There's also a human impact, as the staff wonders what additional debt will be left behind on this project, and anticipates what life will be like with yet one more "completed" project accumulating interest.
As a whole, the key impact is that technical debt undermines agility. For every increase in technical debt, there is a corresponding decrease in agility, just as with borrowing more and more money, eventually you are working to only pay off the interest and not decreasing the principle. Much the same as financial debt takes away options for a family, corporation or government, technical debt does the same to an IT department - more and more resources are committed to paying interest instead of the principal.
Q. Are advances in project management and development methodologies helping to resolve technical debt issues?
A. The reduced cycle time implied by lean architecture and lean software development both help and hurt. On the one hand, IT gets more direct input and feedback on what it has done. But on the other hand, the pace of change is greatly accelerated, leading to more of a temptation to sacrifice and take short-term solutions that might require greater cost in the long run.
We as an industry are learning to manage IT in a more scientific way. Thinking about concepts like technical debt is just one example, another is the Dev Ops movement, with its focus on metrics and cycle time reduction. Simply introducing concepts like technical debt into the minds of those managing IT changes the way IT management thinks, we can recognize when debt is being incurred and make better-informed IT investment decisions as a result.
Q. How can more effective change management help organizations deal with technical debt?
A. As part of the change management process workflow, a specific step can be introduced to analyze tradeoffs not just in terms of the technical merits, but from a business perspective. What amount of debt does one option have versus the other? Change management processes have often focused on risk to the business in terms of downtime or other more concrete impacts, but rarely has there been explicit focus on the accumulation of technical debt. Organizations need to make time in their flow of change requests to pay back prior accumulated debt by favoring related changes over additional features or new systems.
Also, as change requests flow through that would permit debt to be paid back, don't leave them in the backlog. Commit to paying back debt at a rate that will help you incrementally stay afloat.
Q. What advice would you give organizations that frequently end up struggling with technical debt?
A. Just as a credit counselor might tell a person on the verge of bankruptcy:
1) List out all areas where debt has been incurred and figure out which have the highest interest rates, and start paying those back first as part of the change management process.
2) Based on the list from question #2, look at where debt is accumulating the fastest, and put measures in place to stop it (get a handle on new debt).
Ultimately, the key lies on the front and the back end of the change management process - analyze anticipated debt upfront, and make sure it is worth the cost. Then in a post-mortem after the change is completed, re-analyze the actual anticipated debt, and if debt will accumulate too quickly, have the discipline to address it before the interest is racked up. Meanwhile, of course, all systems and processes should be monitored for unanticipated debt, and if deemed unaffordable, that debt should be paid back as soon as possible by putting in a change request.
Technical debt can leave organizations mired in inefficiency as their ongoing IT operations are slowed by shortcuts taken in the past. There are times when debt is necessary to give businesses an edge, but taking on too much debt or taking too long to pay it back can leave an organization vulnerable to operational limitations and fiscal losses. IT service desk solutions can play a vital role in easing technical debt issues by giving IT teams the tools they need to identify frequent problems and manage change more efficiently.
To learn more about technical debt, check out these podcasts by Russ Miller:
- Software Architecture as a Risk and Cost Management Discipline with Eltjo Poort
- Zero-Debt through Continuous Inspection with Brian Chaplin
- ROI Calculations for Architectural Decisions with Rick Kazman