How To Deal With Technical Debt

...is the answer to why your development is slow!

In this issue, we are going to talk about the following:

• What Is Technical Debt?

• Types Of Technical Debt

• Strategies to Fight Technical Debt

• A Recommended Strategy To Deal With Technical Debt

• Tools to track Technical Debt

• Keeping Technical Debt Low

What Is Technical Debt?

Technical debt can cause so much frustration and burnout to development teams. Software engineers can be aware of the side effects of technical debt. However, they often need to explain to the product team why quick and easy solutions to coding development are risky.

So, instead of stabilizing the situation, the business keeps adding more features, and the technical debt grows. We usually say because of this that technical debt is not a technical problem. For that reason, every software development team must try their best to prevent the technical debt from accumulating so it doesn't result in the worst-case scenario, which is a project halt.

The term was coined by Ward Cunningham in 1992. at the OOPSLA conference [1] as a metaphor for developing a software asset. He concluded that the development process leads to new learning, as it depends upon artifacts he coined as technical debt.

For Technical Debt, we can say that it’s everything that stops us from developing software fast. Martin Fowler explains in his technical debt quadrant four different pathways that lead to technical debt creation [4], but there are more. And we can see this, especially with startup companies who want to go fast and don’t think about quality first.

In studies on developer productivity [2], developers are usually forced to introduce a new technical debt as companies keep trading quality for short-term goals, such as new shiny features. Yet Technical Debt doesn’t only impact the whole organization but also developers’ happiness, job satisfaction [3], and morale [8].

The main problem with Technical Debt is that code is an abstract concept, so it’s hard to explain what is happening to business and management. So, companies could easily ignore what they don’t see or understand. So, here we need to be explicit and visualize our Technical Debt clearly for the company and management.

Usually, during the development of our software systems, the capacity available for new features per development cycle decrease (lead time is increasing) while complexity builds up. So, we spend more time fighting the complexity than developing new features.

The image below shows how technical debt is reducing teams' ability to develop new features.

Over time capacity to develop new features decreases

We want a high-performing organization that will give us a competitive advantage to be more responsive and innovative. To do this, we need a high-quality software development process to go fast.

Technical Debt will occur anyway, yet we need a strategy to reduce the creation of new debt while allowing us to reduce existing debt. Investing in this will build technical wealth for our company.

Types Of Technical Debt

We usually think that the bad code = Technical Debt, but there is more to it. There are different types of tech debt:

1. Code quality - our code could be clearer to understand. There are no coding standards, it’s poorly designed, and there is a lot of complexity. In addition, there are a lot of code comments explaining what something does.

2. Testing - we need a proper testing approach (check Test Pyramid), from unit tests, over integration to E2E tests.

3. Coupling - between modules that block each other increases the time to deal with such code.

4. Out-of-date Libraries or Tools - we are using some legacy libraries or tools with several issues, such as security or inability to update to new technologies and platforms. We, as developers, always want to work with the most up-to-date technologies and efficient tools.

5. Manual Process - some processes in our delivery need to be automated. No automated builds, no CI/CD process.

6. Wrong or no architecture - we don’t have proper architecture, or it’s just a big ball of mud. Our architecture doesn’t reflect what we want to achieve with our system, or it doesn’t scale well.

7. Lack of Documentation - there is no documentation, or it’s not updated to reflect the system’s current state.

8. Lack of Knowledge Sharing - we don’t have a culture of knowledge sharing, so it isn’t easy for newcomers to get up to speed. We should always document our decisions and specifications during our work.

And there are more types, too, regarding low-value work, issues with monitoring, etc.

Yet, we often see in many organizations a need for more processes and strategies to manage and reduce technical debt and a sign of needing proper engineering culture.

No time for improvements

Strategies to Fight Technical Debt

So, what are some strategies to reduce technical debt? We should all start by prioritizing Technical Debt. First, we need to analyze our development process and codebase to find bottlenecks and create a Tech Debt Map. Then, once we have such a map (shown in the image below), we need to review each piece by identifying causes.

Now, when we have this list, we can go through it and check what happens if we do nothing (could it become worse or not) and if this part of our system is used for new development now and in the future. So when we mark these two things, we can prioritize and keep stuff with worse tech debt, and we will build on those now and in the future.

Also another alternative is to build this map in terms of effort and pain to fix issues or the RICE Scoring model.

Tech Debt Map

You can also organize a workshop or Tech Debt retros to build a map, and you should regularly audit debt across all of these categories.

A Recommended Strategy To Deal With Technical Debt

One recommended strategy to deal with Technical Debt consists of the following steps:

1. We need to be transparent about our development, bottlenecks, and why we need to go faster. Then, we need to summarize this in a language that allows all stakeholders (engineering and management) to share the same understanding. Then, Here visualizations help a lot.

2. Our systems need to have clear ownership and be responsible entirely for them. But, unfortunately, if some parts of our system aren’t right, no one is taking care of them.

3. We need to prioritize stuff based on impact, as we showed on the Tech Debt Map above.

4. We must empower teams to fix problems and resolve Technical Debt in the natural flow of product development. We need to have a healthy balance here between reducing tech debt versus adding new functionality with a pragmatic approach. What works well is to continuously invest 10-20% of development time into this. How you can do this:

   1. For smaller debt (up to a few hours), do a refactoring when you touch it. But, then, let people be Good Boy Scouts.

   2. For medium debt (from a day to a few days of work), you can try the following:

      1. Tech Debt Fridays are working only on this.

      2. Mark tech debt as a feature, prioritize it, and work on it.

   3. For more significant debt (from a few weeks to a few months):

      1. Take fixed time for tech debt. E.g., take a story and work two days on it.

      2. Have a special task force (dedicated team) to handle it for some time.

5. Use metrics, such as code issues (e.g., by using SonarQube) or lead times. These metrics can help us to make better decisions around fixing Technical Debt.

Tools to track Technical Debt

There is a different kind of tools where you can track Technical Debt. However, I recommend always using some existing tools because introducing new ones can be costly. In general, Technical Debt can be followed in the following ways:

1. Project management tools - the tools such as Jira, Asana, GitHub issues, Azure DevOps Board, or some more specific tools for this, such as Stepsize, CodeScene or NDepend (for .NET).

2. Automated code analysis tools - you can use tools for code analysis, such as SonarQube, to track and detect code issues as well as security issues or other problems.

3. Manual tracking method - here, you can use your manual way, which most fits you, such as Excel table or whatever works for you. It can even be a combination of the above methods, such as tracking strategic Technical Debt in Miro, tracking tactical Debt in Jira, and using metrics from SonarQube. Just make it accessible and easy to update.

   

   CodeScene for tracking Technical Debt (Credits: Adam Tornhill/CodeScene)

Keeping Technical Debt Low

We want to produce code with the right speed, quality, and complexity, not in a quick & dirty way. As software evolves, the architecture of our systems becomes more and more complex, which makes maintenance difficult. Yet if we did a poor design and code quality from the start, we would be in trouble later.

Our goal is to produce systems that are easy to maintain, that we can avoid errors and that issues and further improvements can be added at a low cost. Software architecture and craftsmanship should be appropriately set to achieve this. We want to balance quality, speed, and complexity (as shown in the image below).

Yet, the faster we develop new functionalities, the more it’s hard to ensure software quality, and the tech debt tends to pile up, leading to a slower development process and more errors. So we want to go slower from the start, but this will gain us more speed in the future.

The ideal situation for software development

The question is how to avoid creating tech debt in the first place. The answer is to have a high-quality software development process. And here are some things we can do:

1. Architect systems for the desired outcome. Involve architects from the start in the creation of teams by defining their topology [7] because otherwise, Conway’s Law [5][6] will make our architecture to be an image of our organization’s communication channels, not desired architecture.

2. Write documentation for all your architectural decisions (ADRs) and technical documentation. Make it easy to find (place it in the repo near the code).

3. Write tests on all levels of the Testing Pyramid. Aim at 60-80% of code coverage.

4. Encourage a code review culture. Involve everyone in code reviews and check for code readability, design, performance, security, testability, and documentation. It is one of the best ways to share knowledge.

5. Pair Programming works where one developer writes an ode while another provides feedback and suggestions. This results in higher code quality; with this implemented, we don’t need code reviews.

6. Allow developers to refactor. We should build such a culture that there is no need to ask permission to fix technical debt! Especially if it’s something that can be done in a few hours.

7. Follow codebase standards, such as design, naming, and architecture. Then, automatize this through tools such as linters or your CI/CD process.

8. TDD (Test-Driven Development) is a software design process where each functionality is designed by writing a test first and then doing a proper implementation.

9. Write high-quality code. Educate your developers first to know their programming language in-depth and understand OO concepts, design patterns, refactoring, clean code principles, architecture patterns and styles, and SOLID, YAGNI, KISS, and DRY principles.

10. Continuous Delivery. Code is adjusted in short cycles, so it’s always easy to verify by other team members. This method avoids making significant issues in production by employing frequent releases and fast feedback.

References

[1] “The WyCash Portfolio Management System.” Ward Cunningham, Addendum to the Proceedings of OOPSLA 1992. ACM, 1992.

[2] “Software Developer Productivity Loss Due to Technical Debt, “Besker, T., Martini, A., Bosch, J. (2019).

[3] “Happiness and the Productivity of Software Engineers,” Daniel Graziotin, Fabian Fagerholm, 2019.

[4] “Technical Debt Quadrant,” Martin Fowler, 2009.

[5] “Conway’s Law,” https://en.wikipedia.org/wiki/Conway%27s_law.

[6] “Conway’s Law,” https://martinfowler.com/bliki/ConwaysLaw.html.

[7] “Team Topologies: Organizing Business and Technology Teams for Fast Flow,” Matthew Skelton, Manuel Pais, 2019.

[8] “The influence of Technical Debt on software developer morale,” Terese Besker, Hadi Ghanbari, Antonio Martini, Jan Bosch, Journal of Systems and Software, Elsevier, 2020.

---