#239 - Taming Your Technical Debt: Mastering the Trade-Off Problem - Andrew Brown

[00:01:28] Introduction

Henry Suryawirawan: Hello, guys. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I’m very excited to have Dr. Andrew Richard Brown, or I’ll call him Andrew Brown in this episode. So Andrew is the author of a book titled Taming Your Dragon. Um, so it’s a book about tackling or managing technical debt, which I’m sure all the engineers here, uh, know about technical debt. Uh, we hate technical debt, but also at the same time, sometimes we create technical debt ourselves. So today we are going to learn what is actually technical debt and how can we manage it much, much better from Dr. Andrew. So Andrew, uh, looking forward for this conversation. Thank you so much for this opportunity.

Andrew Brown: Thank you for inviting me on.

[00:02:24] Career Turning Points

Henry Suryawirawan: Right. Andrew, in the very, uh, beginning, I’d always love to invite my guest to share a little bit more about yourself, right? Maybe you can use like career turning points that you can share so that we can learn from you.

Andrew Brown: Yeah, sure. So, um, yeah, so I think that it was perhaps about three career turning points for myself really. So the first one was I did a PhD in semiconductors. And so I had a plan. And my plan was to have a career in semiconductors because I could see that as a bright future. And there was gonna be a big future in semiconductors there. But when I graduated, the semiconductor industry was in recession and it remained in recession for about two years or so. And so I think then that, that basically, that taught me a, an important lesson right at the start of my career, which was that actually, although you may have flat plans, you are gonna have to be flexible about those plans.

And so I looked around and eventually I got another job at a, in a different industry, which was in scientific instruments. And I traveled around the world actually as an engineer installing these machines. So they were pretty big machines. They were called mass spectrometers. They were about the size of a car, weighed a couple of tons, and they cost about half a million pounds, and they took about three months to install. So you’re spending a lot of time in foreign countries there. So I saw a lot of different countries, a lot of different cultures, and that sort of woke you up as well too. Things could be different, equally valid there. And actually that’s where I met my wife or my late wife. Um, I was working in Japan and I then met her whilst I was working out there. And again, that came to another big career decision and I decided to move to Japan so that we could be together. And so I moved to Japan. We had a nice time and then eventually we came back and we lived and settled down in England there.

The other big change or the big career turning point was that when I started, I was in a lot of hardware there. And I resisted. I could see that software was increasing, but I resisted going into it. And part of the reason was that I could see that I knew some friends and they’d been in software development. But the technology had moved on. The languages they were using had gone out of fashion and they were out of a job and they found it very difficult. And so I was very reluctant to get into an industry where they just threw people out of work like that. But actually, eventually, I went in there and I’m glad I did because although, and I have been thrown out of work a couple of times. Actually, so long as you’re keeping your skills up to date, and that’s a really important thing in software. If you keep your skills up to date so you’ve got something marketable, you may lose one job but there will be another job there somewhere that you can get into. And so I would say that that was an important lesson there really of making sure that you always stay up to date and current or you are learning something new. Always, always be learning something new. Yeah.

Henry Suryawirawan: Oh, thank you for sharing your story. I think it’s very insightful and very timely as well in this kind of a time, right, where people, maybe some of us are feeling a little bit concerned about our jobs, our career. Maybe because of the AI disruption, maybe because of the economical situations and so many changes, uh, happening globally.

So I think, uh, I really love the first point as well where you should be flexible. I think many people career, right, doesn’t go as planned in the very beginning, right? We always love to plan, oh, I wanna do this, I wanna stay on this career track, to climb the ladder or something like that. But most of the people that I’ve talked to actually said that their career plan actually are different than what they used to think about in the very beginning.

[00:06:49] The Importance of Skilling Up in Tech

Henry Suryawirawan: So I think maybe, uh, let’s touch on a little bit on this. So for people who are maybe in lots of concerns these days or whatever situation that they are in. So maybe looking back, what kind of, um, confidence, that made you, you know, brave enough to take those kind of, uh, risky moves, maybe back then, right? Because you just said your friends, you know, got out of a job because programming language changed and things like that. So maybe people can get some inspiration from your journey as well.

Andrew Brown: Yeah. So I would, I would say that in terms of, uh, making risky moves, yeah, it’s something that actually people talk about afterwards, but at the time, they’re not actually as risky or you make it as- as un-risky, at least risky as possible. So for example, if AI’s coming along, it would be madness to leave your job and start learning AI or something like that. You should be continuing in what you are, seeing what is happening. And most of us can read fairly clearly that big things are happening in AI. And you can actually then start to learn about AI and get some skills there that are marketable in the environment and in the industry. ‘Cause one of the, you know, terrific things about the internet is that so much knowledge, so much information, firstly, is available, and secondly, so much of it’s available for free. You don’t have to be paying lots and lots of money for these things. You can do it, and there’s people out there that are happy to take your money off you, but there’s enormous amounts that is free. And a lot of it’s from very, very good institutions like the MIT, Massachusetts Institute of Technology, is giving away loads and loads of its online courses for free there. So look for education that’s free and get yourself skilled up before you need to be skilled up.

Henry Suryawirawan: Yeah, I think it’s always, uh, very useful to always keep yourself kind of like hungry for knowledge, right? Keep learning as and when new technology coming in. Uh, definitely we have fears. I also have fear about, you know, how AI is gonna disrupt my role. But I guess, um, depending on your passion, depending on your interests as well, you can always keep that hunger to always learn, upskill yourself and make yourself marketable, that which is something that is very important as well.

[00:09:08] The Definition of Technical Debt

Henry Suryawirawan: So, Andrew, today we are going to talk about your book, Taming Your Dragon: Addressing Technical Debt. So in the first place, maybe let’s go to the definition of technical debt, because many people in the engineering, software engineering industry would have used technical debt as a term, maybe quite loosely. Um, so maybe in the first place, what do you mean by technical debt?

Andrew Brown: Okay, so I think that I would be fairly conventional with what I mean by technical debt. I would take technical debt to be the additional work which you will or someone will need to do in the future as a result of choosing to a solution which is expedient in the short term, but is going to cause you longer term issues there. So, I would say it’s no more than that really.

[00:09:58] The Broken Analogy of Technical Debt as a Financial Debt

Henry Suryawirawan: Wow. This is, uh, quite a very interesting definition because, uh, yeah, you can see that there’s kind of like a trade off here, right? So long-term kind of problems, but a very maybe short, fast delivery, right? Uh, it’s kind of like taking shortcut in a sense. So I think I like that definition, right? So, because many people associate technical debt in software engineering, especially with financial debt.

So, you know, thinking about, okay, we take some debt here maybe for our loans, mortgage, whatever that is, right, and we’ll kind of like repay them back sometime in the future. So I know in your book you mentioned about this kind of analogy maybe is broken in some ways. So maybe tell us why financial debt may be not a good association with technical debt.

Andrew Brown: Yeah, so certainly. I think that, well, the first thing to say about the analogy to financial debt is that it’s given us tremendous benefits there. And because first and foremost, everybody understands financial debt. So everybody has taken out a loan of or owed money in some way, and so they understand that. They understand the benefits and they understand the consequences. So they understand that perhaps if they take on some debt, they can do things that they couldn’t otherwise do. And that’s the benefits of it. So, you know, you, if you, um, you want a job in the next city, but you need a car to get there, then you take out a car loan. And so that’s a good debt to take out. And you also understand that you’re going to have to continue paying that after you’ve received the benefits. It’s very useful in that way because it gets people to understand it very, very well. So it allows people very quickly to understand the essence of it.

The problem with the analogy of technical debt is that it isn’t an exact analogy. And so technical debt is, it would be like you took out a car loan in order to get this job, but you don’t know what your interest rate is, and you don’t know when you’re gonna have to repay it. And maybe you’re never gonna have to repay it.

Maybe somebody else is gonna have to repay it, you know, in which case we’ll take out some more. And you don’t know exactly all these details of it and you don’t know what the interest rate is. And you don’t know the repayments or anything like that. And also, perhaps the most scary thing is that at the same time you’re taking out this technical debt, perhaps someone is handing out lots and lots of credit cards to that account there. And all these other people, including your daughter, spend thrift daughter, she can then go out and rack up lots and lots of debt on this debt to get benefits for herself, which you are gonna have to pay out. So that’s perhaps a closer analogy to what the technical debt really is. It’s a very scary, very strange kind of financial debt there.

[00:12:41] The Role of Human Bias and Organization Issues in Technical Debt

Henry Suryawirawan: Right. So I love that you mentioned that we don’t know the interest rate that is gonna be accumulated if you don’t pay, repay the debt, so to speak, right? And you kinda also don’t know when you should repay the debt fully, right? So I think that’s a very good, uh, reminder, right? Because when we talk about financial debt, usually it comes up with all this commitments, you know, contract, whatever that is, right, upfront. But when we incur technical debt, probably we don’t know how to answer some of those.

[00:13:07] Tech Debt is a Trade-off Problem

Henry Suryawirawan: So you mentioned about, for example, people take technical debt for shortcut. Um, some of these are really not just technical kind of decision, right? But in your book you mentioned it could be organizational issue or some kind of human bias. So tell us these two elements. Why is it also important to actually put in place when you want to understand technical debt?

Andrew Brown: So, I think the first thing to understand about technical debt is that it’s not really a technical problem. It sounds as if it is, because it’s got the word technical in there and it’s got this debt. But really what it is is it’s a problem about trade-offs. So nobody takes on technical debt just for the fun of it. We take on technical debt for a reason. And usually, we take on the technical debt because we want to be able to do something that we couldn’t otherwise do if we didn’t take on that debt. So it’s usually, it’s going to be something like we want additional features there.

So for example, the product manager might, they know that they that this technical debt is gonna be bad for the product long term, but they need to squeeze this extra feature in. Or perhaps it is for a project manager, it is, they know that piling up technical debt is going to be a problem, but they need to get this project in and they are being measured by that. So it’s technical debt is just a consequence of a trade off that people have made. So they’ve made it there. That then leads to an interesting point because practically every company you go into, they will say, we’ve got too much technical debt. So then you’ve really gotta ask, well, what does that mean, you know? It must mean that you’ve made trade-offs that now you are regretting, in which case then you have to ask, well, what has caused you to make trade-offs that you now regret there? And then that gets into the technical debt, the trade-off there. So really, if you want to understand why you’ve got too much technical debt, you’ve really gotta understand how your mind makes trade-offs and how you trade one thing for another and why it is that technical debt comes off so badly in that trade off, really.

[00:15:15] Building a Healthier Relationship with Technical Debt

Henry Suryawirawan: Yeah. So I always find it quite fascinating because I personally also have this kind of thinking, right? So we all hate technical debt. Every time we go to organizations, seeing any kind of code base, we can point out really, really, fast that, okay, this is not good. This design probably doesn’t scale well. It’s a technical debt here and there. But every time we create, for example, new code, develop something new, conscious, either consciously or unconsciously, we are also creating technical debt ourselves.

I think also maybe because this is the nature of software development, where you can’t always, you know, create a perfect design since the very beginning, right? So maybe tell us this kind of, uh, I would say is a paradox, right? Sometimes you hate technical debt, but you’re always incurring it no matter what. What do you think will be a good mindset for software engineers out there to have a healthier relationship with technical debt?

Andrew Brown: I think that really, to have a healthier relationship with technical debt, what they need to be doing is the engineers need to be moving beyond thinking of technical debt as this technical problem there. It’s actually because it’s a trade-off problem, almost certainly, they are only gonna be partially involved in making that trade-off. Often, it’s gonna be the stakeholders. And so the stakeholders, they’re going to want this project in or they’re gonna want these features. And they are prepared to put up with, either put up with this technical debt or they are unaware of the long-term consequences of it. So it’s really, it’s having that conversation and uncovering and surfacing these contradictions there. And basically between the technical debt, balancing technical debt and balancing whatever those business needs are, because if you built software so there was no technical debt, it would be delivered so late that it had no business purpose there. So there has to be some kind of trade-off, and it’s probably, it’s gonna have to be made by the business. But the business aren’t sufficiently well informed about the technical consequences of this that they can make a fully informed decision. So it’s having those conversations there with the business around it so they can do a better balancing act there.

Henry Suryawirawan: Right. I think it’s a very good reminder because, uh, sometimes technical people, software engineers, we love to make decisions solely, purely from technical point of view, right? But I think you just put up a good reminder, uh, because you need to communicate. You need to un-surface, uncover this kind of a potential issues, if let’s say you take a certain decisions. I would say it’s more like an informed trade off rather than, you know, purely just technical basis, uh, kind of like a trade-off. And sometimes also when we develop, we know that it’s going to go into certain direction, uh, and we just assume, many assumption in our head that, okay, this will go this way. But actually, if you talk about it, maybe with your team, maybe with the stakeholders, I mean it could turn out to different direction. Maybe different trade-off will be taken altogether, right?

[00:18:17] The Technical Debt Onion Model

Henry Suryawirawan: So maybe speaking of which, um, for putting a healthier relationship, I know that you also come up with this technical debt onion model, which is kind of like framework or maybe perspectives how we should, I don’t know, like uncover different layers of technical debt. So tell me, uh, well first of all, the idea of this onion model.

Andrew Brown: So the idea about the onion was that, well the first part was I had this realization that technical debt is not a technical problem. It’s this trade-off problem. And then understanding that. But then I gradually, I realized that even if you address the trade-off problem there, so in other words, if you understood how we make trade offs there. And the way that we make trade-offs is, or in fact the way we make any decisions, is we use our emotions. We may think we use logic but actually we use our emotions. And what we use in our logic is it’s just post-decision rationalization. We use the logic to rationalize the decision that our emotions made. In which case then you’ve gotta understand, well, why in that emotional decision does technical debt fared so badly? I look at that in the book then and talk about that.

But then what I realized was that even if you address those problems there and you know that this technical debt is bad, and you know and you understand that emotionally, what happens is that actually you are not making your decision as an isolated individual. You are making your decision as a role in an organization. And in that role, you’ve got some responsibilities, some demands that are made upon you. So if as the project manager you are making that trade-off decision about technical debt, you may know that it’s really bad for the company to take this on. But also you are being measured by getting this project in you’ve gotta getting it in on time. And that’s what you’re being measured by, not by the technical debt. So you are acting in your own best interests, and you are acting actually as the company is pointing you with its incentives and its rewards and its punishments there. So you are, you are doing what is appropriate for that role there. So you need to understand how decisions in systems are made.

So you really need to understand systems thinking as well for two reasons. The first is that these decisions in a system will be made differently from individuals. But secondly, you need to also understand how things work when you have them in a system there and how, for example, by piling up technical debt, you can get a runaway effect there of suddenly too much technical debt causes the development to really slow down from which you cannot recover. So you really need to understand these systems as well. And then what I realized from there was actually, if you’re talking about individuals making decisions within a system, essentially that’s their economics. That’s what economics is. And so then I looked at, well, what can economists teach us?

And economists have been looking at problems similar to this for hundreds of years. So they’ve got a whole series of problems that they have highlighted and addressed. And the really good thing about looking at it from the economist point of view is that, firstly, that they, they’ve studied it already, so you don’t need to reinvent the wheel. But secondly, they are talking the language of business very often. So very often we, we find it difficult to engage with the business. And we may talk about technical debt and we may talk about these risks and this and that. But all the business people here is ya-ya-ya-ya.. And it doesn’t really go in. Whereas if you start talking about externalities or the principal-agent problem there, it’s likely that it’s something that they met at business school. And so straightaway, you are talking their language. So that’s an advantage of this, looking at the economics layer.

And then underneath that I put a final layer, which was around wicked problems. And so really, technical debt, and in fact all, every, practically everything that we do in software development is a wicked problem. And so I talk about wicked problems and tame problems.

And so tame problem is it’s a problem like chess or a game of some sort that has some rules and there is a single solution or there’s a solution which everyone will agree is better than another solution there. Whereas in a wicked problem, it’s got several characteristics. Uh, one of which is that different people may have different opinions about what is the best solution there. So for example, on that of the road I live on, perhaps there’s a bit of noise. And my best solution is that there’s no noise at night. Whereas for the young people that are coming back from the pub, there’s a better solution which is that there’s a bit of noise as they come back from the pub. And… but it’s all quiet by three o’clock. And so you can see that’s one thing that you could have there is that there’s different people may have different opinions about what the solution is.

But another thing as well, which is very highly relevant for software development, more so than any, most other types of engineering problems, is that a wicked problem, very often, you will not know the, um, you will not know the solution until you find the solution. So you won’t understand the problem until you develop the solution. And very often for us, we don’t know what the software is going to be because it’s part of the exploration. We think we are building one thing and it ends up we tell people we are building something else, and in the end it ends up as something entirely different. Whereas if you looked at say, civil engineering, say constructing a bridge, you’re pretty sure what you’re gonna end up with before you even start. But that’s not true in software development. So very often we, we don’t understand the problem until we got to a solution. So that’s sort of a part of this wicked, wickedness of the world that we live in there.

Henry Suryawirawan: Wow, thanks for kind of like walkthrough of all the layers. Just to recap, right, so you have kind of like five layers. It’s kind of like an onion model, right? Layers by layers. The inside, maybe the core, maybe it’s technical layer. And then you have the trade-off layer. You have the systems layer, kind of like the systems thinking aspect that you mentioned. Then you have the economics of game theory layer, and the last one is the wicked problems layer. So maybe let’s try to peel it slightly, um, smaller layer, right? So maybe I won’t go deep into technical layer because I think for software engineers, it’s kind of like assumed we know what kind of technical problems. Maybe the design, scale, you know, lack of tests and all that. We’ll just skip it all together.

[00:25:10] The Onion Model: Trade-Off Layer

Henry Suryawirawan: But let’s go to the trade off layer, because I think this is quite important as an individual, right? Whenever we make decisions, we sometimes make a lot of trade-off, either knowingly or unknowingly. So what are some of the interesting aspects of the trade off layer that you want to remind us or you wanna kind of like advise us today?

Andrew Brown: So essentially, yeah. So the trade-off layer, when you’re making trade-offs, it’s important to fully understand how you make trade-offs, which I said is we use our emotions. And although we might talk about logic and rationality, most of the time the rationality comes in afterwards, and we just use it to justify the decision that we’ve made. And so you’ve really gotta understand what it is, how you make decisions using emotions. Essentially you use something called the effect heuristic, which is, it’s like you’re a sort of, it’s like your- an emotional sort of gauge there. And what that does is it guides you in the decisions that you make.

And there’s several important things to realize. One of which is that this process is entirely subconscious. So you don’t understand why you make these, um, decisions. Although you will be able to justify it, you’ll be able to give a good reason, but that reason will be, uh, somewhat spurious. And so they’ve done quite a lot of experiments on people making decisions or having preferences. So one of which was, um, they took some, I would call them kanji characters, but that’s because I’m coming from a Japanese background there. But it would be the Chinese characters, and they would flash this, uh, they’d have this Chinese character up on the screen. This is to Western people who don’t, who cannot read, uh, the Chinese characters. And they would ask them to make a preference. Which one do you prefer?

Now what they didn’t know was that although they were looking at a screen there, it was a special screen and it had a very fast refresh rate. And so what they were doing was they were putting up onto the screen, every so often they were putting up a picture of a happy face or an angry face. And what people did was they had a preference for this character, and they would give a reason why. Because it was a, you know, it had smooth curves or it did this or did that. But what it actually was, was that behind their flickering too fast for them to see, there was a happy face. They didn’t like the other one because it had an angry face, but they never realized this because it was so fast. So our emotions or our decision making, it happens so fast that it’s outside of our consciousness, which makes it very difficult then to control really. But we do need to be able to control it and we can control it.

And an example here would be, so there’s a, there’s an analogy which is quite close to technical debt. So technical debt, it’s you’re trading off between a feature and taking the debt on. So the characteristics of the te- the feature are, it’s gonna be a media. You’re gonna get the feature as soon as it’s coded, it’s gonna be concrete. You get the feature or you don’t. So it’s gonna be those things there. Immediate, it’s gonna be, uh, concrete there. Whereas if you look at the technical debt it’s gonna, potentially you’re gonna have to repay some debt in the future. But you don’t know when, you don’t know how much, you don’t even know what it’s gonna be in there. It’s very indefinite. It’s very abstract there. And so you can see that something that’s concrete and immediate, it’s gonna appeal to your emotions. Something that is in the future and indefinite and in abstract, it’s gonna appeal to your logic, but not your emotions. And so that’s principally why the technical debt does badly.

And a useful analogy is to smoking. So in smoking, most of our governments, they realized several, many decades ago that smoking is bad for the populations. They wanted to stop it. And so one of the things they did was they decided to put, um, health warnings on the cigarette packets. And the, um, the tobacco manufacturers, they fiercely resisted this, and they fought it through the courts, but eventually they had to put the warnings on there. But they didn’t need to worry because what happened was, if you look and you read that warning, it appeals to your logic. It’s talking about a potential future hazard to your health, which may or may not happen in the future. And that appeals to your logic, whereas if you looked at the cigarette packets, you had packets of pictures of sexy cowboys and things like that. So as a teenager, that was appealing to your emotions. And so they put these warnings on that and it had very little effect. Whereas if you look now at a, in certainly in our country, I’m not sure about yours, you cannot get packets of cigarette like that. They’re all plain branded, so the manufacturers cannot affect your emotions. What they’ve done in our country is they’ve put horrific photographs of damaged body parts, damaged by smoking, which of course, appeals very strongly to your emotions there. And so now it’s been switched over that the health authorities, they are controlling their message through the emotions. And it’s not the only reason, but it’s part of the reason that smoking rates have gone down in many countries there.

So that’s like a clue of what you need to be doing with technical debt. If you’ve got, or you want to change the level of that technical debt, you need to make an arguments that are appealing to people’s emotions, not to their logic. I talk a bit about this in the book where I say there’s several ways you can do it. So one is you can sort of paint a picture. You can tell a story and you could put together a scenario of you wanted to do some development, but you’re prevented from doing so by this technical debt there. So that would be one thing there.

There’s other things as well you can do to affect the way that you make decisions. So one is something called a Ulysses contract. And a Ulysses contract, that’s where you commit to something that you cannot later reverse that commitment. They’re used quite a lot in, um, they’re like living wills. So if you come to the end of your life and you are losing your faculties, you can put in this Ulysses contract so that at a certain stage you won’t be able to reverse those decisions there, and somebody else will be making those decisions. So you can put in a Ulysses contract for, say, technical debt where a project agrees to take on this technical debt, but if they do that, then they have to apportion part of the project, uh, funding to addressing that debt at a later stage. So there’s a few things that you can do like that, yeah.

Henry Suryawirawan: Yeah, so I think it’s very, um. Interesting the way you explain about these kind of a trade-off analysis, right? Because sometimes we think it’s very kind of like simplistic, uh, mindset, right? Like we think, oh, we take this debt simply because we want to go faster. But sometimes it’s more beyond that, right? So you mentioned about emotions, right? It could be maybe also that day when you make the decision, you kind of like not in a good mood and sometimes you take a shortcut simply because, I dunno, like you just wanna finish things faster. Could also be the trade-off that we mentioned in the very beginning, right, with business, right? Business always wants, you know, things to get delivered faster, many features. Project deadline that gets set without, you know, being asked from the engineers, right? So those kind of things definitely make trade-offs, um, kind of like more complex than what we think about when we take technical debt.

[00:33:03] The Ulysses Contract for Managing Technical Debt

Henry Suryawirawan: And another thing you also mentioned about kind of a few solutions, right, that we can do. Maybe painting the pictures, uh, trying to kind of like put the kind of like the story. If let’s say we take too much technical debt, what would happen? Maybe in the business terms could be, yeah, we couldn’t scale beyond x number of users, uh, or we couldn’t process transactions, uh, you know, beyond whatever number of RPS and things like that. And Ulysses contract I think is very interesting. I’m not sure like how many people have that in their day-to-day team, uh, kind of like contract, right? So maybe in your life exp, yeah, in your life experience, have you seen this Ulysses contract kind of like implemented?

Andrew Brown: I’ve advocated for it, but at the organizations where I’ve advocated and put it forwards, it hasn’t been taken on there. But yeah, so I think, yeah, I think like most of the time when I’ve been advocating, I’ve probably been a bit too late. We’ve already starting to hit crisis point, at which point it really doesn’t matter what you suggest, it is just gonna have to try and get out the door and worry about everything later, yeah.

Henry Suryawirawan: So maybe one implementation of Ulysses contract that I can think some teams actually implement. So maybe it’s, for example, three sprints of, you know, business technical features more, and then you spend one sprint just to focus on technical improvements, technical debt. That could be a simple contract that any teams, I believe could actually implement in their day to day.

Andrew Brown: Yes, that-that, yep. That’s a way of looking at a Ulysses contract. But in some ways you’d need to be a little, you’d also could be a bit more granular in that if you get two solutions, right, we can… and I can think here of when I used to work at HMV. There, we had a particular problem in that we wanted to display a product in two places at the same time, which is understandable. And basically in order to do that, to do it properly was a big, big rewrite of the database. And we couldn’t possibly do it before Christmas. And Christmas was really important for HMV, because about a quarter of all of their sales are just at Christmas time. So you are never allowed to do anything that messes up Christmas for HMV. And so what we worked out was there was a quick and dirty solution, which was we could create a duplicate of the database so you could put this product in two places at once. But then you had a lot of maintenance. So anything that needed to be updated or you had any feeds in, all had to be duplicated. The company with the business agreed yes, we’ll do the quick and dirty. We’d get it in before Christmas, and the first project that we will do after Christmas is we will undo this and we’ll do it properly. I’d like to say that’s what happened, but it isn’t. The first project is when we came to it actually, we realized that so many code fixes had gone in during the Christmas period there, that all of a sudden it was much, much more difficult to actually go back and redo it properly. So they decided not to do it and they were doing any future project and they kept on putting it back until eventually I left the company and it had all got it had all gone on. Yeah.

Henry Suryawirawan: Right. Yeah, I was about to say sometimes, yeah, we got this, I dunno, like promise, right, in the very beginning. Yeah. We will let you do, you know, like repay your technical debt and all that, but sometimes, uh, because of whatever business situations, we may never get it fully repaid after all, right ? Thanks for sharing that personal story.

[00:36:32] The Onion Model: Systems Layer

Henry Suryawirawan: So let’s move on to the next layer, which is, uh, you call this systems layer. So you kind of like associate this with systems thinking a little bit. So what interesting thing that you wanna share about systems thinking or maybe systems layer that could actually affect technical debt?

Andrew Brown: I would say. So, yeah, I mean, so in terms of systems, in software development and IT, we’re actually starting from an advantage, because we already have a fairly good understanding of systems. Because practically every project that we work on is within a system. And if we are putting in a big project, say like an ERP system or something that’s really big, almost certainly there’ll be one or several people looking at the problem and looking at it from a systems point of view. So we’ve got that understanding there of systems. So we have an intuitive or it’s more than an intuitive. We have a specific, um, and a concrete understanding of systems. But really that’s IT systems and that’s like physical systems there.

As well as the physical system, which is important, there’s also the social system that we are working within, and that’s where these decisions are made. And so what you’ll see, an important thing about social systems or living systems, not just social systems, is that different parts of the subsystem, they can develop their own sub-goals and they can pursue their own sub goals, which may be in conflict with the goals of the whole organization there. And that’s something you don’t normally get that in physical systems because the designers will have… you won’t get in a well-designed physical system. And mostly you won’t get it because it’s been thought about. Whereas when we’re putting in or we’re working in software development, the testers, they can have their goals. The software developments, they may have their goals, and of course, business will have very different goals there. And they can be clashing there. And so this can lead to unexpected developments there.

And so I mean the, one of the most famous ones is you’ve probably heard of Prohibition in the United States. The banning of alcohol is about, about a hundred back, it was about a hundred years ago. Beginning of the 19, of the 20th century, a lot of countries had a lot of problems around increasing rates of alcohol consumption. There’s various reasons for that. People were getting a lot more money. They were working in factories, they were getting better paid and they were going out and drinking. That’s what happened. But this was causing lots and lots of social problems. And so different governments tried to deal with it in different ways.

And the United States, they decided they were going to use prohibition, which was just ban all alcohol. But what they hadn’t realized was that, or they hadn’t considered the system inside which the alcohol is sold and consumed there. And so they banned alcohol. And so all the bars closed down. All those breweries and distilleries, they all closed down. Everything. Everything like that stopped. But you still had people that they used to be drinking and they want to continue drinking. And so anybody who’s done economics, economics 101 is gonna tell you if you have a demand for something and you suddenly cut off the supply, there’s gonna be a substitute supply from somewhere. And this substitute supply came from criminal organizations. It actually, it came from criminals who actually then became organized because they weren’t particularly well organized before that.

So then it, this led to lots and lots of problems in Prohibition. So you had a rising crime, a lot of violence. You had enormous wealth being generated for these criminals who then, they would then use it to bribe the police, bribe the courts, whole series of things like this. And they built up these very, very strong organizations there. And so one of the longest lasting effect of Prohibition in the United States was something that was entirely unexpected or unanticipated and highly undesirable, which was the growth of organized crime in the United States. In our country, we tackled it in a different way and we didn’t get those gangs building up because of that.

Henry Suryawirawan: Wow. Quite fascinating story, right? So I think many of us could relate with this kind of like social system aspect, uh, especially if we work in a bigger organization, right? So things could be also like political reasons, hierarchies, right? Or silo teams that work, you know, quite independently without talking to each other. It could be so many other things like incentive system, reward system, punishment system, right? So those kind of things, maybe from one small perspective point of view, it seems could solve a particular problem, right? But unconsciously, you can create unprecedented effect in some other parts of the system, right? So I think this is where systems thinking kind of a mindset is very, very important, especially for us who are working in software engineering. So never kind of like focus or invest too much on one particular solution, but thinking holistically how it could affect the other parts of the systems as well, I think, is very important.

[00:41:50] The Onion Model: Economics/Game-Theory Layer

Henry Suryawirawan: So yeah, let’s move on, yeah, to the next layer, like economics game theory. You mentioned about using this model so that we can communicate with the business much better. Um, maybe some of examples of economics or system, uh, or game theory layer here that you could advocate for us to use next time we talk to our stakeholders?

Andrew Brown: I would say in, right, in terms of that layer there, there’s lots of very fascinating things in there, and it’s very tempting and very easy to become completely embroiled in it and look at some fascinating aspect of it. But I’d say perhaps the, two, if there were, if I was to choose two things, it would be, firstly, you don’t need to communicate this to the bosses, but… Actually, it wouldn’t hurt to communicate it, but it’s the principal-agent problem, which is that it’s a problem where basically the principle so that the boss or whoever has the money, you are paying or rewarding somebody else to do something of your bidding there. And that’s essentially when we’re working in the company, we are the agents and then the organization itself is the principal.

And so if you look at the principal-agent problem, what you will have is that, they may have different goals and they may have different motivations there. And so what you should always be aware of and thinking about is how do their goals line up? You’re going to be most successful if the principal’s and the agent’s goals line up or coincide there. If, and you should also understand that you’ll be unsuccessful to the extent that those goals don’t line up there. And it’s of course, it’s not just the employees, but you’ve got an added complication in most organizations in that they will often, they will bring an external help to manage large projects. And what you have to realize is that perhaps the goals of those outside organizations, the consultancies, may not be very well aligned with the organization that they’re working for.

And certainly, I’ve been on a lot of projects - I imagine you have as well - and you will have seen some of the antics and the activities that some of these consultants will get up to which, and not necessarily helpful for the client there. And here I’m thinking of things like perhaps rivalry between organizations there. Another one might be that actually one of the things that perhaps some of the less ethical consultancies might do is as soon as they go in, is to try and remove all of the capability of the organization so that the organization is now dependent upon the consultancy. So the whole series of things like that. So there’s this principal-agent problem there, which I think people should be aware of there. That’s gonna play out in the technical debt there of that the company may want lower, low levels of technical debt. But the principal, if the principal is the, if the project manager who is going to be perhaps moving to a different organization at the end of the project and their goal is to get the project in or peer in, and it doesn’t matter to them how much technical debt there is, you’ve got that principal-agent problem there. So I’d say there’s the principal agent problem that is there.

And a second thing that I think that I would also highlight to the business, because they won’t be so well aware of this, is the problem of externalities. So an externality is, basically it’s where one party can impose a cost or a benefit. But usually it’s a cost. You don’t worry about benefits. They can impose a cost on another party and the other party has no say whatever in whether they accept that cost or not. And the most commonly cited example is pollution. If you drive your car into the city and you are there perhaps to pick up your wife and she’s out shopping, and so you sat on the side of the road there waiting for it to come out the shop there, And you had your engine idling. Well, you are creating all this externalities in terms of the pollution that you’re creating. But you’re seen in your car and you are not suffering that externality. It’s others that are suffering it there.

And that’s very much what happens on projects there, where projects may be making decisions around how much technical debt they are piling up. But they’re not necessarily gonna be the ones that pay off that debt. Perhaps it’s gonna be the support team. I mean, this was brought home to me once when I was at, um, or not once, many, many times, irritatingly many times when I worked at HMV. We had a project manager there. And whenever he caused a problem that affected a problem, that affected another project or support, he would just turn around and someone said, oh, right, you, what you’ve done now has done this. And there he would turn around and he would say, that’s not my problem. And say, he was just saying effectively, he was saying, I am putting an externality on you. And the structure of our organization is, I don’t care whether I do that or not. And the company doesn’t care and I’m getting away with it.

And technical debt is very much, very often it’s an externality. ‘Cause the people that are making the decisions about it very often are not the ones that are gonna end up paying that back at the end there. So those are the two things that I think, that I would take from economics to technical debt. Principal-agent and externalities.

Henry Suryawirawan: Wow, I think now that you are kind of like explaining and giving examples, right? I’m sure some of us kind of like laughing, you know, hearing those examples because we could actually realize it’s happening to all of us, uh, in our career, right? So I think definitely this kind of principal-agent problem, right? Externalities where someone makes decision without actually taking the responsibilities later on is kind of like quite common, especially in software projects where you kind of like engage consultants, third parties, or project basis rather than product basis, right? So I think, uh, we can see it in real life.

[00:48:10] The Onion Model: Wicked Problem Layer

Henry Suryawirawan: And let’s go to the next one, which is the wicked problem. You kind of like, um, touch on a little bit earlier, right? In that wicked problem is something that maybe you don’t even know in the very beginning, what could the solution look like? You can always explore, experiment until the solution actually comes out. This is so much evident in software, right? Because we’ll iterate, right? So we’ll come up with a very simplistic design that could maybe solve the problem, the features. Show it to people, people will ask for more, and then we continue evolve it until it gets to a different kind of solution altogether. So why do you think wicked problem is kind like the outer layer, which I assume is gonna be the most complex thing to consider. Um, so what kind of insightful thing can you share with us today about wicked problem?

Andrew Brown: The wicked problem, I would say that the… There’s multiple parts to the wicked problem as you are saying there. So one characteristic of a wicked problem is that you don’t know what the problem is until you actually have a solution. Other things are that there’s no right or wrong, there’s just better or worse. But perhaps the really critical thing for us is that different parties will have different beliefs about what the problem is and what the better solution is. So we may think something from a technical side, uh, there that, well, this is the best solution. But from a business point of view, they may be thinking something very different there. And it’s how you resolve those things or attempt to resolve them or accommodate them there. That’s something which I think most of us will find very difficult, particularly within software development, because we tend to be very technical people and we tend not to think of things in terms of political, political sort of landscape and how we need to accommodate different things there.

And really the people that perhaps are best equipped to deal and understand those, that political landscape and so on, they’re all sitting in the business there. It’s almost as if we need to get them or someone from that side to come over and advocate for ourselves there. Otherwise the risk is that we end up in just like a standoff there, of the technical people saying this technical debt is bad and we have too much of it and we need to get rid of it. And the business saying, yeah, but we, we are getting do- the market is being dominated by somebody else, we need to get a bigger market share which means we need more features or we need more this or more that. And once we’ve got a bigger market share, we can sort out all these other things later. It’s almost as if you’re having this argument but you are arguing in two different languages there. And I think that’s tremendously difficult and wasteful thing to do there.

And I think that it’s not just for technical debt. I think there’s a whole series of soft things around here or we have a whole lot of problems in software development, which are wicked problems there. And I think we should un, we should perhaps try and understand better about wicked problems and what we can indeed do about them there.

Henry Suryawirawan: I think, uh, after you explain it, I’m sure, um, you know, people can relate, right, what do you mean by wicked problem in software engineering and why you put it as the outer layer simply because, yeah, it’s very difficult. Um, sometimes you could wear multiple hats. Only then you could see where things are kind of like not aligning, you know, things are always miscommunicated, misunderstood. And sometimes, yeah, like you mentioned, people don’t have the same definition of problem, doesn’t have the same definition of solution, right? Or maybe we don’t know the solution in the very beginning, uh, only until, you know, we reach a certain point. So definitely software engineering, I find it’s a lot of wicked problem kind of a thing.

[00:52:03] How Organizations Can Start Managing Technical Debt Better

Henry Suryawirawan: And we can see from all these layers that you just explained, right? Technical aspect is just a small part of the technical debt. And there are so many other things like organizational system, the bias thinking, the emotion, social structure and things like that. How would you advise software engineering team or organization to start managing their technical debt better or at least introduce technical debt in a more healthy manner rather than, you know, like trying to burn as many technical debt as possible and then try to repay in the end which will not happen. And in the end, you know, it cause a lot of problems, or a lot of damage in some of the stories you share in the book. So maybe how can organizations start managing their technical debt better?

Andrew Brown: I’d say almost certainly if you are managing technical, if you’re coming at it from the viewpoint of we’ve got this managed, this technical debt, and we need to manage it or burn it down, you are almost starting too late, really. It’s what you really need to be doing is avoiding creating the technical debt that you can avoid creating there. And so I think that really there is, it’s almost like, uh, an analogy would be doing a development at a sustainable pace there. You can’t do development really, really fast. And then when you become completely tired and knackered, now we will slow down to a sustainable rate. It’s a bit too late, you know. So it’s getting people into the mindset that they are making these trade-offs and these trade-offs will be there for an awful long time. And getting them to make better trade-offs there.

Now there’s always going to be technical debt, and they need to, you need to get this tech- you need to take on this technical debt. If you didn’t, you couldn’t get the products out there in time, you’d be too late in the marketplace. But you don’t need to take on all the technical debt that you’re taking on. And some of it would be unnecessary there. And in particular, I would say it’s things such as when you’re trading off technical debt for additional features there, you really need someone to push back on do we really need these features? Because it’s very easy for the business and the marketing to say, this is absolutely essential. This feature, we must have it in. And no one can prove otherwise. And so it’s just because it’s so easy, it’s so tempting to do that again and again and again, until you end up with this very bloated product with lots and lots of debt in it. So it’s kind of starting at the beginning there.

And one of the things that I talk about in my book, in the last third of the book there, is really how you can address the technical debt. But it’s addressing the technical debt from the mindset of avoiding creating it in the start there rather than getting rid and burning down this technical debt. In fact, that’s a very bad thing to do in many ways because once you do that, it’s almost as if you’ve given the business permission to create this technical debt because you can just pay it down later. Try and get them into this mindset of avoiding this technical debt and the costs of this technical debt there and why and how they can avoid it or minimize the amount of technical debt there. Yeah.

Henry Suryawirawan: Wow, I think it’s a very good, uh, reminder, right? So, uh, whenever we talk about managing technical debt, probably it’s a bit too late, right? So I think avoiding it, prevention is always the best cure, right, rather than taking pills, medicine or adding people, buying software, whatever that is, right, to actually kind of like solve the technical debt problem.

And I think you mentioned a very good keyword that I think we all need to always reflect, right? Sustainable pace over a long term, right? So the moment you feel like, kind of like the velocity slowing down, the team cannot deliver things as smooth as before, right? Maybe that’s, uh, kind of like also a signal for us to kind of like think about how we can tackle or manage our technical debt better and communicate that to the business stakeholders, right? And in your book, you also mentioned a few common anti-patterns. Uh, I think people who would love to kind of like advocate or champion, you know, managing technical debt better in an organization you should check out Andrew’s book.

[00:56:16] The AI Impact on Technical Debt

Henry Suryawirawan: So it will be amiss if I didn’t, you know, mention or discuss about AI with technical debt, right? I think with the advent of AI, we can certainly see there are many aspects of technical debt that could be, I dunno, kind of like remodeled, right? So the first aspect is like with a lot of AI churning out code, I think many people say it could create a lot of technical debt, right? Or maybe also, and some other people say, oh, we have a very high technical debt system. Now with AI, we can use it to actually solve technical debt. Maybe simply because, I don’t know, at least documenting the feature because the people have left or things like that. Or creating tests automatically. So what do you view, uh, about AI? Um, is it something that is going to increase technical debt or is it gonna be reducing technical debt?

Andrew Brown: I think that the most prominent there and that the most salient, visible part is the big increase which is gonna come from non-technical people doing vibe code development there and creating enormous amounts of technical debt there. So I can see that being a huge, a huge thing there. But I could also see that it may be that certain types of technical debt may become less important, particularly if you are able to create this coding and this solution so much more easily. It may well be that the debt is less important perhaps. But I would say certainly it’s the AI and the vibe coding does give you a very strong feeling that this is something that could create enormous amounts of technical debt. Particularly since I imagine vibe coding being written, and if something is useful and it’s taking off, it’s probably not gonna get rewritten. It’s let’s build on top of that there. In which case, yeah, you’re building on something that’s, uh, probably got a lot of debt or a lot of limitations in it, yeah.

Henry Suryawirawan: Yeah, so there’s this danger, obviously about vibe coding or you just blindly accept whatever AI suggests, right? And I think I wanna touch on also like the emotion aspect, right? Whenever we use AI, sometimes we feel very confident and it’s very easy, right? We could just churn out code after code after code, building features on top of features. Sometimes there’s, I think there’s a dopamine hit as well whenever you build such thing rapidly and you could see results. You almost see the results, you kind of like polish it more and more. Uh, but unconsciously you introduce a lot more technical debt. Do you have like real case experience or examples where people use AI effectively? Like cautiously also not to take on more debt, but also kind of like build a system that can also last like much, much longer.

Andrew Brown: Well, okay. So I’ve spoken to people that have been using AI, and what I’ve… Of course, what they’re telling you may not be exactly the same as what they’re doing, of course. But what they were saying was that they were using it quite a lot for investigation and research there. And if they wanted to find out something, they were perhaps posing a question to an AI or an LLM model there. They were getting the answer back, they were looking at it, they were then refining their question, firing it back again, and going through a cycle then multiple times. But then often at the end they were manually coding. They were doing it there. So they were doing it for this research there. That, at least that’s, or that’s what they were telling me. Of course, what people tell you isn’t always the same there. And perhaps there’d be a reluctance to say, yeah, I just let AI do all of the stuff for me there. I, guess we will see. The results will come through I think. And we’ll be seeing those surely I think.

Henry Suryawirawan: Right. So I’m pretty sure, uh, one day we can, because now everything is moving, everything is changing. Uh, and AI technology itself is being kind of like reinvented, you know. Every week, every month, you know, maybe there’s a new model coming, new techniques coming out. Agentic thing now is also kind of like the crazy thing. And I’m sure AI will pop up in your layers. I would assume it would be in the trade-off layer and sometimes also in the wicked problem, right? So I think, uh, definitely it’s gonna be fascinating this, uh, AI trend going in the software development world.

[01:00:35] 3 Tech Lead Wisdom

Henry Suryawirawan: So, Andrew, uh, as we reach the end of our conversation, thank you so much for outlining about this technical debt. I find it very fascinating for us, educating also, uh, for us, like how we can make technical debt trade-off better. Before I let you go, I have one last question. Uh, it’s like a tradition in my podcast, which is to ask this question called the three technical leadership wisdom. So if you can think of it just like three advice you want to give to the listeners, uh, maybe if you can share your version today, that would be great.

Andrew Brown: Yeah, sure. So, well, I would say, um, that basically read more deeply, okay. If you think about where you’re spending your time, quite a lot of your time, it might be you are just reading something on, uh, LinkedIn or perhaps watching something on YouTube. There are some really wonderful things that have been written out there, and here I’m thinking of, published books, okay? Something that is published and something that is a book, someone has spent a lot of time thinking about that, doing that. They will have put in hundreds or thousands of hours to produce that. So there’s hundreds or thousands of hours that have gone into that thought, and you can get that thought for an investment of just, uh, you know, 10-20 hours or something reading through it there.

That’s not always gonna be true though. If it’s, say, a blog article there, it won’t have been through all of that process there. But if it’s been published and it’s been published by a publisher that has got… They need to make money. And in order to make money, they need to do something that’s good enough quality that people will read it there. And there’s a lot of stuff out there. And here I was thinking about, um, you know, uh, one of your, more recent guests was, uh, Karl Wiegers with his Programming Pearls. And he’s like, what a wonderful book there, you know, with all of these lots of good ideas there. And, you know, similar, there’s one with, um, uh, it’s Lessons Learned in Software Testing. Lots and lots of really great ideas in there. I think developers, we probably read perhaps two books a year or something, two technical books a year, if that. We could- we really should be reading an awful lot more. That’d be the first thing, reading in our own subjects.

But then I would also say you should also, you should read more widely. And here I’m taking a sort of a hint from something called red teams here, which is where you should read outside of your experience there. So, I mean, I do a lot of, a lot of work and a lot of research into the human factors. But the human factors that had not, you know, I had nothing at all to do with it until I started thinking about why have we got all these problems in software development that have been around for decades that we haven’t. And so I started looking around for where else have they had similar things. So, you know, I’ve looked at that and I’ve looked at systems, and systems thinking and dynamics. So you should broaden out your, um, your ideas and read more widely. And particularly, you should also read things that you don’t necessarily agree with. You should expose yourself to different points of view.

Now this means though, then this comes to the third bit here. You need to set some time, you need to allocate time to do this. So put a slice into your, uh, calendar there. Maybe it’s gonna be a morning or something every fortnight, just for reading and nothing else.

Henry Suryawirawan: Wow. I think, uh, that’s a very, again, very good timely reminder for some of us who, you know. Because these days, social media or maybe short blog posts are so much easy to access and sometimes we get notified as well. I find sometimes like all these creates fatigue as well, right? So you have so many content thrown out to you. So we kind of like spend maybe short time reading it. We forget a lot of things. I think you, your advice is very good, timely reminder, right? Read more deeply, read more widely as well, and set the time to actually do those two things right? I think for software engineers, yeah, I also think myself don’t read enough, especially those technical books that could be very thick, uh, and sometimes very dry. Rather than entertainment.

Andrew Brown: Yes. Yeah. Yeah. That is true, yeah.

Henry Suryawirawan: But I think it’s very important, especially if we wanna keep ourself up to date. Re-skilling, again, like coming back to the first thing that we discussed, uh, in our conversation.

So, Andrew, if people love this conversation, they wanna reach out to you, ask you more questions, maybe about technical debt or social behavioral, uh, aspects, which I know that you’re passionate about as well, is there a place where they can find you online?

Andrew Brown: There’s LinkedIn. You can probably find me there, Andrew Brown at LinkedIn. Or I’m on YouTube as well. Or you can fire me an email which is. We can perhaps put it in your session there. It’s Brown, Brown Sensei is the email to get me at brownsensei@hotmail. I took brownsensei when I was working in Japan actually at the time. Yeah. And yeah, I’d love to hear from people.

Henry Suryawirawan: Right. Very nice, uh, nickname, Brown-sensei. So I think, uh, I’m sure like we all learn a lot of things from you today from the Sensei, so thanks for sharing your knowledge and your experience. So yeah, thank you again Dr. Andrew.

Andrew Brown: Thank you. Thank you very much, Henry. Thank you. It’s been a pleasure. Thank you.

– End –