#100 - Modern Software Engineering - Dave Farley

[00:01:32] Episode Introduction

Henry Suryawirawan: Hello to all of you, my friends and listeners. Welcome to the special episode of the Tech Lead Journal podcast, the show where you can learn about technical leadership and excellence from my conversations with great thought leaders in the tech industry. And this is the episode number 100. Oh my God, I can’t believe that I finally reached this special milestone! In the beginning, when I started my journey, creating this podcast, I made a pledge to myself that I would reach 100 episodes. And here we are, two years later from my very first episode. I’m really proud to have reached this milestone, and I could not have done it without the support from each of you who listen to the episodes week in week out, and also special thanks to all the guests who appeared on the show and for sharing your expertise and wisdom! I really, really appreciate your trust, collaboration, and for your willingness to share back to the community. And personally, I have grown a lot in the past two years by serving all these great contents, and I really hope that all of you enjoy and grow yourselves as well.

And by the way, if this is your first time listening to Tech Lead Journal, make sure you subscribe and follow the show on your favorite podcast app and the social media on LinkedIn, Twitter, and Instagram. And for any of you who wants to contribute back to the creation of this podcast, support my journey by subscribing as a patron at techleadjournal.dev/patron.

For today’s special episode, I am super excited to share my conversation with another great thought leader in the software industry. Since I read his groundbreaking book, “Continuous Delivery”, and applied many of the principles and techniques mentioned in the book, I have benefited a lot in my professional career and still continue to advocate its best practices wherever I can. The book itself won the Jolt award in 2011 and has been continuously impacting the industry since then, even until now. And so as many of you would have already guessed by now, my guest for today’s special episode is Dave Farley. Apart from co-authoring “Continuous Delivery” book, these days, Dave also runs a highly successful and popular “Continuous Delivery” YouTube channel on software engineering topics, which I also watch and learn from it a lot.

In this episode, we discussed Dave’s latest book, “Modern Software Engineering”. Dave started by explaining his view on modern software engineering and why it emphasizes on practices for building better software faster. Dave described the foundations of the software engineering discipline, and he explained the core competencies we need to succeed in modern software engineering by becoming experts at both learning and managing complexity. Dave also explained the importance of understanding technology fundamentals, improving software readability, and handling software complexity by managing concurrency and coupling. Towards the end, Dave also shared some other tools in the modern software engineering toolkit, that, of course, include Continuous Delivery.

I really enjoyed my conversation with Dave, learning the fundamentals of modern software engineering, and importantly, the practices that we can use for optimizing learning and managing complexity. If you also enjoy listening to this episode, please share it with your friends and colleagues who you think may also benefit from listening to this episode. It is my ultimate mission to spread this podcast to more people, and I need your help to support me towards fulfilling my mission. Before we continue to the conversation, let’s hear some words from our sponsor.

[00:07:07] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I’m very excited. So I got the chance to meet someone who I follow a lot by reading his book, following his blogs, and also, recently, his YouTube channel. His name is David Farley, also more widely known as Dave Farley. I mean, just to share, when I read the Continuous Delivery book, this was like many years back, I find that book is one of the cornerstone of my career. Trying to implement those practices, like delivery pipelines, Continuous Integration. I really find the book is something that is really critical for my journey. So I really thank you for that, for writing the book. And as an ex-Thoughtworker as well, I always love to see all the ThoughtWorkers as an author because they write a very insightful book based on experience. Thank you so much for spending your time today, Dave. Looking forward for this conversation.

Dave Farley: It’s a pleasure. Thank you very much for asking me. And I’m pleased that you enjoyed my book.

[00:08:01] Career Journey

Henry Suryawirawan: So in the beginning, maybe if you can share a little bit of background about yourself. Maybe telling us your career journey that maybe people haven’t heard about. Any highlights or turning point.

Dave Farley: Sure. So, as you can probably tell by looking at me, I’ve had quite a long career. I’ve been doing software development for a very long time. I’m quite old. I started off probably at the start of what was then called the microcomputer revolution. So the move from kind of mini computers to computers that were more accessible based on Silicon integrated circuits that were starting to become cheaper. So I had the advantage of kind of the first computers being fairly simple and me kind of growing up with them over time, which was pretty nice, really. I’ve done quite a lot of different things. I spent the early part of my career working for hardware manufacturers. I worked for computer manufacturers and writing operating systems and device drivers and those kinds of things.

And then later I got interested, I don’t know, by accident, but in kind of distributed systems. Starting to build, for those days, larger scale distributed systems. I think I have a shag for kind of being part of the team that invented the concept of microservices a long time before they actually took off. So we kind of invented them and then they went away again. We did some really interesting stuff for a research company. Building software with these very loosely coupled, we called them collaborative business objects. It’s still the most interesting software in the sense that you could introduce two pieces of independently developed software and they could do useful work together, which I’ve not seen anything do that quite as well. It was a fascinating project to work on.

One of the seminal moments for me was, as you mentioned, working for ThoughtWorks. I worked for ThoughtWorks and I’d already been doing, informally, some agile things. I’d been doing a little bit of eXtreme Programming in projects before that, but ThoughtWorks kind of turned the volume up on that for me. And we started doing what we thought at the time, and I think the one most people credit us with is working on some of the biggest agile projects that had been done up to that point. And so we will start pushing the boundaries and finding difficult things, difficult problems. We’ve had to apply eXtreme Programming when you’ve got 200 programmers. We started addressing that kind of thing, and I think that was one of the things that sort of gave birth to the concepts that have steered my career since in terms of Continuous Delivery, and taking a more engineering kind of approach to thinking about solving these kinds of problems.

The latter part of my career, I spent nearly all of my career building real software for people one way or another. About six years ago, my friends and family convinced me that I ought to start my own business. Cause I was always a bit nervous of doing that kind of thing. And so, I started a consultancy these days. My job really is a software consultant. Advising organizations all over the world, usually big organizations on how to do a better job with software development.

[00:10:48] YouTube Channel

Henry Suryawirawan: And not to mention also producing YouTube content because you seem to be very active in churning out contents every week.

Dave Farley: Sorry to interrupt you. That was an accident as well. As you can probably tell, I’m not somebody that plans my career. That was an accident forced on us by the pandemic. I was very busy with consultancy. The pandemic happened, and that stopped. My wife and I were looking around, saying, “What are we going to do?” We’re supposed to go to Brazil next week. What are we going to do there? And so we started the YouTube channel. Partly just as a means of keeping us sane and having something to do. And it’s been extraordinarily more successful than we ever anticipated or hoped, which is very nice.

Henry Suryawirawan: I can also see the plaque behind you. So I think that kind of like after you reached some hundred thousand subscribers or something like that, right?

Dave Farley: That’s right. So we got that earlier this year. We just passed the 120,000 subscribers. We’re rapidly approaching, or we just crossed the 5 million views mark. So it’s quite a big channel now for a technical channel. We released a video every week on a Wednesday, and we get tens of thousands of views every week. We usually would get about a quarter of million views a month. That’s quite a lot of people coming along with the journey and at least listening to these ideas, even if they don’t always agree with them, which is good. Cause maybe we can have the debates.

Henry Suryawirawan: And it’s not just about the number of viewers, but I think the quality of the content, I think, is pretty rare still. I rarely find YouTube videos that cover all these technical contents that is deep, and also based on real experience. So thank you so much for producing that and I hope that it will continue to grow.

[00:12:19] Modern Software Engineering

Henry Suryawirawan: For today, I think we will discuss mainly from your latest book, which is titled “Modern Software Engineering”. The first thing that comes to mind for me, when I look at the title, why do you call it Modern Software Engineering? Is there the old ways of software engineering and there’s a new modern way of software engineering?

Dave Farley: Yeah. It was a title that was stuck in my mind for a long time while I was writing the book. And to be honest, I was somewhat nervous at the time. The reason that I call it modern software engineering rather than just software engineering, and I was nervous of the title is kind of the same. I think in our industry, we’ve kind of weirdly devalued what we think of as engineering. I think it’s very common to think for people to say software development isn’t engineering. And a lot of it isn’t. I think that’s fair. A lot of it isn’t engineering. But in other disciplines, engineering doesn’t mean heavyweight bureaucracy, which I think is what it tends to mean to us.

What engineering means, you know, the discipline, is the most effective, efficient way of doing work of high quality. And so if software engineering doesn’t allow us to build better software faster, it isn’t software engineering. So modern software engineering, I think, is different to what it was before. We tried to implement engineering approaches before, and they were nearly always narrowly focused on some particular set of tools or diagramming techniques or something like that, and bureaucratic and heavyweight. And that’s not what I mean at all. So I mean almost entirely the opposite of that. So what I was interested in, parts of my background and interest, my hobbies is that I’m very interested in science. I think a reasonable definition of engineering is as a practical implementation of science. You apply scientific style reasoning, which is humanity’s best approach to learning, to solving practical problems. And I think that’s a reasonable kind of colloquial definition of engineering.

I was always interested in those kinds of ideas. What are the ideas that we could point to and whatever your technology, whatever the problem you are solving, whatever organization that you’re working in, if you applied these principles, you’d have a better chance of success. And I thought I’d spotted some of those things over time. And then related to Continuous Delivery, which was my previous book that I worked on with Jez Humble and so on, but also somewhat separate in a slightly different angle. It’s kind of orthogonal. They intersect and they’re very reinforcing of one another, but they’re not the same idea. So what were the principles that would allow us to do a better job? And I thought that was what I was trying to explore, and that’s what the title was meant to try and convey.

[00:14:58] Better Software Faster

Henry Suryawirawan: The subtitle of the book itself is “doing what works to build better software faster”. And in your book, I also find it quite insightful is that you mentioned that if our software development practices do not allow us to build better software faster, we should really change them because they are not engineering. So please tell us more about this concept. Why do you think so? Like why we should focus on building better software faster?

Dave Farley: The simple answer is why would anybody want to build worse software slower? I think it’s the combination. In part, this is my kind of, again, colloquial take on the DORA metrics, the metrics that come out to the State of DevOps report that give us a predictive model. If you work in these kinds of ways and do these kinds of things, you increase your chances of success.

And they’re based on the stability metrics. What’s the quality of our software? When we introduce software into production, how often does it break? And when it breaks, how long does it take us to recover from the breakage? And throughput. What’s the efficiency with which we can deliver software of that quality? So that’s the faster part. So, working quickly, one of the findings from the DORA metrics, which is important, I think, is that there is no tradeoff between speed and quality. One of the traditional assumptions is that if you want to build high-quality things, you need to go slowly. What the DORA reports says, “No! If you go slowly, you build lower quality things.” And that’s what the statistics say. That’s what a sociological approach to measuring the performance of software development teams finds. The more barriers you put into being able to release change quickly, the worst the output.

So we want to be able to move quickly. We want to be able to go faster and we want to be able to build better software. And that means that we need to be able to focus and be able to learn, find out where our mistakes are, correct them really quickly and efficiently. And that kind of is one of the ways which is applied with Continuous Delivery. Continuous Delivery is a fantastic way of being able to build a better software faster.

Henry Suryawirawan: From my career itself, I can tell that so many experiences from my view is that we build software faster in the beginning. But turns out after a long period of time or maybe few months, it starts to get slower and slower and maybe many people have left. They don’t know about the context, the domain problems. So I think maybe some of the tools that we will discuss today can help people to really build better software faster, like continuously.

[00:17:22] Software Engineering

Henry Suryawirawan: So you have a definition about software engineering, right? If I can read here, is that the application of empirical scientific approach, which you mentioned in the beginning just now, to finding efficient economic solutions, so the economic solution here is also quite interesting, to practical problems in software. I know that you cover in the book that some of these keywords are really important. Maybe if you can discuss slightly some of these terms. Why do you think it’s important to define software engineering using this way?

Dave Farley: So I was trying to get the simplest minimal description of what I was thinking in terms of engineering. I was quite pleased with that description that you just read out because it’s fairly lean, it’s fairly concise, but I think it doesn’t miss anything important.

So applying the reasoning, the rational approach of science. I think that one of the fundamental natures of science is that the difference is between science and almost everything else is that we start out assuming that we’re probably wrong. And that’s a very healthy thing to do in engineering and in software development at all. So we’re going to apply scientific style thinking in a lightweight, not overburdening, not bureaucratic way, but we’re going to be informed by that kind of thing. We’re applying that to solving practical problems. We’re not doing theoretical research and proving that quarks are the basics of particles or anything. We’re building things, and so we’ve got to be practical and pragmatic. So the empiricism of engineering, just trying stuff out, ultimately, it’s a big part of it as well.

The economic part is, again, part of engineering. If we are physicists, we can imagine building machines that are black holes. Thinking about that practically, economically in energy terms, we’re not ready to do that kind of thing. For us engineers, we are building things, and so there are always economic constraints at some level. That’s about the amount of time and effort and money that we spend. But also economic constraints in the sense of the performance of our delivery system and the performance of that software. The carbon footprint of the software that we create. That kind of thing. It’s an interesting question in its own right. So I wanted to get that in there as well. So I think all of these different parts of the definition kind of lead us in this direction of trying to organize our thinking in a more rational way and apply it to solving problems that are of value to people in some way. That’s what I was trying to reach for with the definition.

Henry Suryawirawan: Thanks for that explanation about the software engineering. So I think for us, we need to probably redefine some of our understandings of engineering. It’s not just writing code that works and deploys it.

Dave Farley: I think that’s a really good point. One of the things that I’m obsessed with at the moment is watching SpaceX build their space rockets to get to Mars. That’s engineering live on YouTube. You can kind of watch it happening in the real world. That’s not theoretical. They’re evaluating their ideas. They’re blowing spaceships off. They’re learning from that. They’re making progress step by step. And I think all of these things kind of coming to an engineering approach to solving problems and thinking about the ways in which stuff can go wrong and all of the time looking out to improve and refine our designs and our solutions so that they’ll better fit the problem that we’re trying to solve.

[00:20:37] Expert at Learning

Henry Suryawirawan: For us to follow this definition of software engineering, you mentioned there are two things that we have to be expert at. So one is actually to become expert at learning. This comes back probably to the scientific approach that you mentioned. The second is experts at managing complexity. Maybe if we can cover one by one. Let’s start with experts at learning. So you mentioned there are a few key things, like five things that we need to be able to practice in our software engineering or software development day-to-day. So what are those five things?

Dave Farley: Yes. So experts at learning. Let’s just say why that’s important first of all. It’s to do with this kind of scientific engineering philosophy of assuming that we’re probably wrong. We’re not going to start out with perfect vision. We’re not going to be able to predict the business context, the technical context, the socio-technical context of how we organize ourselves to work. We don’t know that results in any of those things. At some level, we’re going to get all of those things wrong. And so, we want the ability to make progress while we don’t know at the start of the project when we don’t have all of the answers yet. Therefore, we want to optimize to be able to learn.

Microsoft did some research into their own ideas and found that two-thirds of their ideas produce zero or negative value for the company. So you’d like to be able to learn which were the bad ideas early on if you wanted to be efficient. So we want to optimize to be able to learn all sorts of things. What the problem is that we’re really trying to solve? What our users really need to be able to solve that problem? Which tech solution fits? Does it really work? Is it releasable? Is it compliant? All those things. You want to learn ways of doing that. How are we going to work?

In order to be able to do that, as you said, there are five principles that underpin our ability to do that. So we need to work iteratively. We would need to work in many small steps so that we have many opportunities to kind of look at what it is that we are doing and refine it. “Oh, that didn’t work. Let’s not do that anymore.” That kind of thing.

We need to take feedback seriously. We need to understand what information we need to gather in order to be able to determine whether our ideas or our solutions are useful or helpful, whether our products are landing with our customers, whether our tests are passing or failing. We want to optimize great feedback to do that.

We want to also be able to work incrementally. We want to be able to build systems almost evolutionarily. So it’s a bit by bit. Not all in one great big chunk. So we want to be able to deal with these things more independently so that we can kind of carve it out. And certainly, if we’re working on a big scale, divide up the work so that different parts of the organization can work on different parts of the problem. So working incrementally is part of all of that.

We want to work experimentally. If we want to start more scientific, being more like engineers in our approach, then we should apply a little bit of discipline around the way in which we think about making changes. The key to that for me is to think experimentally. We want to make little predictions. We want to say, well, this is the problem that we’re looking at the moment. We think if we did this thing, this would improve that position on this problem. How would we tell whether that thing was or wasn’t working? That’s the difference between an experiment. It’s about step of predicting what the outcome is, and then carrying out the test. So you could think of this as a test with a product. We think that if we do this thing, it’s going to improve the sales for this widget, and we’ll put it into production to see what happens. Or you could think of it as a test. I think that the code should do this thing. If it were to do this thing, this is the assertion that I would be able to make. Then I write some code to fulfill that assertion. These are ways of working experimentally, and that can, and for me, has become pervasive. That’s the way that I now think about and practice nearly all change where I can. So what is it that I’m trying to do? How would I understand the results of this thing? And I’m just thinking in that way. It doesn’t need to be heavyweight, to be really fast, really short, really efficient, but it’s just a slightly more disciplined, organized way of structuring our thinking that gives us better results.

And the last one is empirical. We’ve already talked about the empiricism. Engineering is about real world things. It’s not about some ivory tower imagination of a system. It’s about the reality of it. So we need to gather feedback from production. We need to be able to understand what’s really going on. We need to be able to make practical decisions based on what it is that we find. Elon Musk is building his space rockets out in Texas. I’m sure that they’re doing very sophisticated computer simulation, but at some point, they build stuff out of metal and then they blow it up on a test and see how it works. That’s how engineers think, and that’s how they operate.

If you look at the development of any sophisticated engine. Think about an aeroplane. Aeroplanes used to be incredibly dangerous things. In 2017, the equivalent of two-thirds of the population of the planet flew in commercial airliners and not one person died. That wasn’t possible in the early versions of aeroplanes. We had to go through that pain of things breaking and killing people, and we say, “Oh, well, we won’t do that anymore. We will learn.” So engineering is about working incrementally and growing our learning based on mistakes that we make fundamentally. And we’ve got to give ourselves the freedom to be able to operate that way and to make those kinds of mistakes booked so when they occur to learn from them and to correct them.

So I think those five things: iterative working, feedback, incrementalism, experimentation, and empiricism are the cornerstone of our ability, really as a species, to be able to learn effectively and efficiently. I can’t imagine anything that we could consider to be engineering without those things. So the first third of the book is really about exploring those ideas in much more detail. You’re looking at each one of those principles and kind of picking apart and seeing how it fits with software development.

[00:26:34] Why Agile Not Enough

Henry Suryawirawan: When I read these five key principles, they seem to interrelate with each other. Of course, they support each other and then you become expert at learning. So these days, many people, I think, already understand Agile. They work iteratively. They probably try to build a short feedback loop and things like that. Where does the thing go wrong? Why do you think still, you still need to reiterate these kinds of concepts? Because Agile, if I understand correctly from the beginning, from the get go is actually to adopt all these kinds of things, right? Iterative, incremental, short feedback loop, and there’s also an experimentation part where after the sprint, maybe iteration that you deploy and learn from the customers. So, why it seems like this kind of things are still forgotten or maybe misunderstood?

Dave Farley: I’ve had the good fortune during the course of my career to be fairly close to the birth of some influential, big ideas. My observation is in all of those circumstances, if you talk to the people who came up with the big ideas after their ideas are successful, they said, “We didn’t mean that”. Because there’s a kind of this semantic diffusion things. People come up with ideas and then other people interpret them differently. And I think that’s definitely happened to Agile. I think if you look at the Agile Manifesto, which is the closest thing to a definition of what Agile really means, Scrum is near, the Agile Manifesto says what Agile is really about. If you look at that, that’s not going to date. That’s still pretty good. That’s a pretty good specification of what goes on.

The problem with agility as an approach, and I’m a card carrying agile person, I’m a believer that it was an important step, but the problem with it is that part of it is about giving people freedom to make their own choices and learn things, and that’s deeply important. But there are naive ways of taking that. And one of the naive interpretations of that is that everybody has a veto on everything. So if you’re working on a team and your team decides that they’re going to do test-driven development, and one person on the team decides that they don’t want to do test-driven development, they won’t do it. They’ll veto it. That’s not really what agile was saying. The level at which autonomy works, I would say, is kind of at the level of a team. So the team agreed they’re going to work this way, and then you do it. Whether you like it or not, you do it. I’ve definitely been there. Parts of the team that were doing things that I didn’t think were the right things to do, but we still did it as a team because that’s what we’d agreed to do.

So I think there are some things like that. There’s also the natural resistance of a big organization to want to change. That’s I think my interpretation. Over the years, I’ve worked with a lot of different organizations, usually bigger organizations, but a lot of them. I think my overall summary, if I could lump them, it’s very unfair to lump them all together, but if I was to lump them all together, if I was to put words in their mouth, what they all want is that we’d like to carry on working exactly as we always did before, but get different results. And that’s not going to happen. So the trouble with Agile ways of working is that they’re quite radically challenging to traditional corporate culture. So they’re very difficult for big organizations to adopt because it means breaking down barriers in organizations.

Without sounding too immodest, I think that Continuous Delivery is currently state of the art for software development. It’s what most of the best companies in the world at software development do, and it works. We’ve got data that says that it works often in certain measures, orders of magnitude better than other ways of doing things. And I think that one of the advantages of Continuous Delivery is it’s just a little bit more prescriptive. It still gives people the freedom to innovate, but it’s fairly hard line on some things. So, part of the definition that I use is if you can’t produce something releasable every day, it’s not Continuous Delivery. That’s pretty hard line. That’s a pretty definite statement. That’s more definite than say, as important as it is, but people over process, which the Agile Manifesto says is a really important idea, but it’s a bit vague compared to produce software, releasable software every day. That’s a pretty hard line statement. A little bit more constrained, a little bit more specificity in what we should be doing, I think sometimes frees us and helps us to be more innovative, to be more creative in some of the things that we do.

I think of Continuous Delivery as a second generation Agile discipline. I think the second generation Agile approaches are more effective than the first-generation one. So Continuous Delivery is also second generation. Extreme Programming for me was the best Agile disciplinary to the first generation. But Continuous Delivery adds a little bit more stuff around eXtreme Programming and takes it a little bit further. But both of those are engineering approaches and that’s the other thing that is a failure of Agile. Agile got subverted to become a project management approach rather than an engineering discipline. Ultimately, we’re building software so we’ve got to be able to create things. However good your planning is, if you’re not actually building the stuff, it’s not working out.

Henry Suryawirawan: So I kind of like your definition of Continuous Delivery is like a second generation of Agile or maybe XP, right? Because some of these practices, it’s like a rigorous process. You have deployment pipelines, running tests, test-driven development and all that. I think that kind of like gives people guidance as to what practices that they need to follow.

[00:31:49] Expert at Managing Complexity

Henry Suryawirawan: You mentioned that if you can not release software every day, so it’s a really high bar, especially for people who started with a good idea in mind. And it seems like very farfetched, really high and big ideas. How can people do this? While at the same time, I think one of the discussion point in the book, you mentioned about cost of change. So if you’re doing traditional software development, maybe it gets faster in the beginning, but over the time it gets slower. But using this modern software engineering approach, you can actually make the flat cost of change to the software development. So tell us more how we can actually aspire to reach that level where, over the time, basically, any change that you make to the software, it’s kind of like flat. It’s not like a steep curve going upwards.

Dave Farley: I think, actually, that brings in the kind of the second section of the book, which is what I call managing complexity. The techniques of managing complexity are really about addressing that problem. So I think that one of the most important qualifying characteristics of what we would think of as quality in software is our ability to change it.

So, sure. It has to do the job that it was designed and intended for you. If performance is an issue, we’d like it to be fast enough and all those kinds of things. But fundamentally, if we’re living in this world where we don’t know all of the answers on day one, we can’t perfectly predict the future and the business direction, then inevitably, we must be able to change minds and change our software. The business direction might change. A solution might not be good enough in some way or another. Or we spot a new idea that’s much better than the idea that we had in the first place, so we want to follow that. All of these are reasons to change our software. So I think that one of the cornerstones of high quality is our ability to make those changes quickly, efficiently, easily. Flattening that cost of change curve, as you mentioned.

For that, we need to be able to work a little bit defensively in the way in which we approach and design our software. So it’s reinforced by the practices of learning, if we’re building our software so that we can iterate quickly on it and we can gather good feedback from it, and we can carry that little experiment. Each change to it is a little experiment of some form that reinforces stability. But then, we need to build software that’s modular, that’s cohesive. The bits that are separate should be separate in the software. The bits that are closely related should be close together in the software. We want it to have good separation of concerns. Each piece of the software to be focused on doing one part of the job. And then, when we need to change our mind on that part of the job, we can just work in that part of the software. We want our software to have lines of abstraction. We want it to hide information between one part of the software and another, so we can make change in one place without worrying too much about how other places work. Having a coupling between the software is a problem and abstraction will give us that. And that gets to the last point, which is to manage the coupling effectively, but with a preference, a tendency more in the direction of loose coupling. So we’d like the pieces of our software not to care too much about the detail level of the pieces.

If we build software that looks like that, then it is much easier to change. If we want to build software like that, then one of the techniques that you can use that massively amplifies all of those five properties, test-driven development prefers software that’s modular, cohesive, separation of concerns, abstract, and loosely coupled. It’s much easier to write test-driven tests for software that looks like that if you have those properties in the code. So we’ve got this nice feedback loop. This nice reinforcing process. We want these properties as hallmarks of quality in our code, and we have these techniques that tend to make us put these properties in the code.

That’s kind of what I mean by engineering. It’s not an automatic recipe. It’s not going to automatically generate great software. You still need to be good. You still need to be intelligent. You still need to be thoughtful. But when you are good, intelligent, and thoughtful and use these techniques, it guarantees you a higher probability of a better outcome. It’s not going to make sure that you’re going to be successful, but you’ve got a higher probability that you will be if you work those ways, I think what the data says and what I’m trying to describe in the book.

[00:36:01] Importance of Fundamentals

Henry Suryawirawan: So when you mentioned all these five key principles: cohesion, modularity, loose couplings, separation of concern. Looking back in the university, we kind of like taught about this maybe in programming 101 or things like that. But it’s kind of funny, like why do we need to keep on repeating this sort of fundamental, basic stuff in software engineering, probably, or computer science? You mentioned the industry currently has a tendency to focus a lot more about tools, technologies, frameworks, or languages, rather than actually emphasizing a lot on design. So tell us, how do we get wrong here? So many people talking about technologies like Cloud, Kubernetes, mobile, whatever that is. But why people de-emphasize on design?

Dave Farley: It’s a good question, and I wish I knew the real answer. I can give you my opinions as to some of the reasons. I think to a degree, it’s inevitable. Software development as a career appeals to certain kinds of people. Those of us that it does appeal to are interested in how computers work and the tools. I’m as likely as anybody else to have a vigorous debate about the merits of different programming languages in a pub with anybody else. But ultimately, it doesn’t really matter very much.

The last language where I thought that I’d kind of consciously work to use every last part of the language was C many years ago. There’s not many parts to C, so you can see I’m a bear of small brand. But as part of the book, I tried to count up how many languages I’ve used in professional settings over the course of my career. And I reckon I’ve probably programmed commercial software, one way or another, in about 20 different languages. So they are ephemeral over the course of 40 years. That’s one language every couple of years. So you might be more stable than me. You might be looking at that more. I’ve certainly done more stuff in Java than I have in Prolog.

But it is interesting thinking about the way that Java works, differences in syntax between Java and Python and all that kind of stuff. We can talk about those things. But fundamentally, when I start to think about it, doesn’t have any impact on the way that I design software, really. It has some nuance in terms of whether I’m idiomatic in the language that I’m using or not. But fundamentally, the way that I organize and carve up the problems that I’m trying to solve with software hasn’t changed very much for decades. In some ways, we find around the edges, the way that I think about it is, I think, a little bit better now. But the same things that were important doing things in C or even in assembler, are still important doing it in Python or JavaScript or F# or whatever else. Those fundamentals don’t change. Modularity matters. Whatever the technology that you’re using, to apply cohesion, the same separation of concern, is the same.

As you said, we learned about these things in university, and then we discard them and say, “How do I do this with Helm charts?” That stuff doesn’t really matter. As a result, what we tend to do, so we look at something like Kubernetes. Kubernetes is a fantastic tool. If you’re Google scale, doing all of these different services, that’s fantastic. I see teams all of the time that were building, forgive me, some very simple little application that’s probably a single process running somewhere, and they start off by deploying it with Kubernetes. And you go, “Why does Kubernetes help for this simple little problem? How does that work?” So thinking in terms of what the problem is that we need to solve. Applying things like modularity, cohesion, separation of concerns tells us how to solve it, and then we figure out the tools to use to apply to maximize our ability to build modular software that’s cohesive with good separation of concerns. That’s the goal. The technology is just the means. It’s like a carpenter being obsessed with the nails that they use and the hammers rather than what it is that they’re building. We need to focus on what it is that we’re building, not the nails and the hammer.

Henry Suryawirawan: Thanks for giving your insights. Like you mentioned, any good programmers, doesn’t matter what language, maybe they can learn new language or even use old language, they can still produce software that resembles all these five key elements. Modularity, separation of concerns, and things like that.

[00:40:04] Importance of Readability

Henry Suryawirawan: The importance of managing complexity, I resonate with this as well, because the more you have a growing complexity code base, the cost of maintaining that software is really huge. And we probably wouldn’t be able to make change as easy as maybe the first time it was written. In the last few episodes of the podcast, I’ve talked with some of the guests about managing code complexity, readability, and all that. I think this kind of ties back to all these principles of design. So why do you think it’s very important to have code that is manageable in terms of complexity and also readable? You mentioned it’s about communication. But this is counterintuitive to people because people think writing software is just to make a program work or to solve a business problem.

Dave Farley: Yeah, and that’s certainly true, but it’s never won up. So the sustaining our ability to solve those problems is in everybody’s interest. It’s in the interest of the organizations that employ us, as well as our own. And I think that’s one of the mistakes that, as software developers on teams, we tend to be too short-term in our thinking, and we tend to optimize next week, rather than next month. If you optimize the next month, that’s some different things. It’s a bit like deciding that the time that you spent putting petrol in your car or charging your electric car up is a waste of time. And so you’re going to deal without that and very quickly, you’re going to run out of gas and your electric car will come grinding to a halt.

If you do that thing, there are certain things that you need to do in order to maintain your ability to make progress. In software terms, that’s precisely what you said at the start, which is, the big problem of building complex software is that over time, it gets harder and harder to change it. That’s because we didn’t do a good enough job of managing the complexity as we added new things. So focusing all of the time on managing the complexity flattens that cost of change curve, retains that freedom to change our minds and to do new things, add new features, modify the features that are already there more easily. It might be difficult work sometimes. But we still have the ability to do that than if we build some big, complicated, coupled ball of mud that we’re afraid to touch.

So managing the complexity is about trying to organize our thinking in ways that compartmentalize the problem. It’s a bit like writing a book. If I wrote a book that was just one long sentence, it might have all of the same words in it, but it’d be much, much harder to read than if I break it down into sentences and paragraphs and chapters and sections and label them. That gives us markers of where we are. We can jump to apply different parts in the book and look at only that part in the book we meant to refer back to. Those are book level tools for managing complexity. Ideas like modularity, cohesion, separation of concerns, abstraction and coupling are software levels of ideas for managing complexity.

To be fair, I apply this not just to software, but to information systems in general. The way that I organize my hard disk is kind of similar in terms of modularity and cohesion. I’m trying to break things into pieces and keep things that are related close together and things that are unrelated far apart on my disk. Cause it makes it easier for me to navigate the system. So I think these ideas are foundational principles that we should apply pervasively everywhere. I’ve been talking to some people building packaged software and low code software. These principles still apply there. I’ve been talking to people that do machine learning. These principles still apply there. I think those are the kinds of things that we would expect if we were to identify genuinely important principles for an engineering discipline that was related to software.

Henry Suryawirawan: Yeah. I like the way that these five key principles can be actually taken, not just from low-level code, but actually to something bigger like machine learning. And sometimes getting these distributed systems. I know that you cover about why microservice exists. Why microservice exists is partly also by adopting all these key principles.

[00:43:57] Concurrency & Coupling

Henry Suryawirawan: There are two things when we build a complex, maybe distributed systems, right? You mentioned that two things that probably is root cause of more complexity are about concurrency and coupling. Sometimes these are unconscious. So when we build software, we don’t think about multi concurrent process at one go and also coupling. It could be dependency coupling, coupling between classes or even coupling with other services. Maybe if you can touch on these two primary causes of complexity in the modern software?

Dave Farley: Yeah, with pleasure. I think those two things, independently but certainly combined, are kind of our equivalent of quantum physics. They’re incredibly complicated, and we are always just moments away from them, which is interesting. As soon as you create a branch in your version control system, or spawn a new thread, you’ve just jumped into that world of complexity, of concurrency and coupling between those concurrent systems. And it always comes at a much bigger cost than not doing that.

Let’s go back to the analogy of writing a book. If you and I are collaborating on writing a book, then we could divide up the work and you write one chapter and I write another, or if we’re working on the same piece of text, we could do stuff and we could send us changes between each other and all that kind of stuff. Or we could put it on Google Doc and we could work on it concurrently and we could see our changes. That last one is a much better way of working and much more efficient way of working because we can see what’s going on and we get that fast feedback. We can learn really quickly. “Oh, you’re working on that part. Henry’s working on that part. I’ll avoid that for now while he’s working on it. Whatever it is, so we get much better feedback those ways. So thinking about those kinds of ideas and how the information works is important.

In the late part of my career, when I was building systems, rather than talking about building systems, I was involved in building what we think was probably the world’s highest performance financial exchange, and that was fascinating. We had to do some, not necessarily me, but the team as a whole, had to do some really very clever things in terms of deep concurrency and managing the flow of information to be able to get the levels of performance that we managed to achieve. We were looking at the specific architecture of Intel processes and designing software that would take advantage of that to maximize the throughput of our code. That’s a very difficult stunt, and it’s the same difficult stuff, but we have, as soon as we spawn a new thread that needs to join up with another thread. Because we did some benchmarking kind of things to look at this. If you just increment a number, and you say, “Oh, I’m incrementing this number to a very large value. So I’ll speed it up by making it to do in parallel.” If you just do that, then the cost of the most efficient way of synchronizing the results back so that you get a valid result at the end is 300 times slower than just doing it on a single thread.

So, there are naive ways of optimizing things, and there are clever ways of optimizing things. Parallelism is very valuable for certain kinds of problems and very stupid for other kinds of problems. One of the places where this surfaces at a higher level, recognizable place is in version control. If you have a relatively coupled system and you divide that system up into a bunch of microservices, each living in a separate repository, but you don’t trust them enough to be able to release them independently, so you’ve got to test them all together before you’re safe to release, that’s one of the more inefficient ways of organizing them. If you put them all into one big repository, there are different kinds of optimizations that you can do than if they’re in separate repositories. You’re adding a bunch of overheads to your ability to go in and change code in different repos all of the time, so it slows you down. There are lots of advantages to microservices that they will give you. But this is a problem of concurrency and coupling between the disk. I think that concurrency and coupling is the hardest parts of software, computer science, really. Everything else is relatively straightforward in comparison to those two things. So being defensive about concurrency and coupling is a very wise strategy.

A couple of my friends are considered to be world-class experts in concurrent systems. Their advice is always don’t do concurrency unless you must. They are the world’s experts. They say don’t do it unless you must. So I think that being defensive about those sorts of things and managing those things. So one of the things that we did at LMAX when we built out very high performance exchange was that we architecturally isolated concurrency, so that didn’t happen where we were doing the business logic. So the business logic for our system, these very, very high throughput, very widely used by hundreds of thousands of users. All the business logic was processed on a single thread for each piece of business logic. But we did smart things in being able to bring the information to the point at which those single threads could process the logic. So we externalized, separated the concerns of being able to do that. So I think there are smart ways of doing this and organizing these kinds of things. But I think concurrency and coupling are the enemy and we should be doing everything else to kind of guard ourselves against those things. That’s part of the engineering discipline, parts of managing complexity.

Henry Suryawirawan: Thanks for sharing these insights. I’m always amazed when in your book you mentioned about LMAX. This is like a very high state-of-the-art kind of system. Some of the principles also that you mentioned is it’s not really just optimizing for speed, parallelizing and all that. Your code must be readable first, and you isolate the concurrencies such that you can still make changes and still the system is performant. It’s not like optimizing syntax or lines of code or try to make assembly code better.

Dave Farley: Absolutely. A hundred percent. One of the things that we learned and measured at LMAX was that readable code is nearly always more efficient than not. Because it’s not that it’s an overhead. It’s because if it becomes readable, you’re going to be striving to make it simple. So it’s expressive and clear what’s going on. And compilers love that shit, because if the code’s readable to us, then it’s readable to compiler. Therefore, the compiler could go to town and optimize it more effectively. But more than that, if it’s readable, it’s going to be simpler. And so there are a few fewer places there for nasty problems to hide.

If you think about what high-performance software really is, it’s about delivering the most impact for the fewest instruction. So that’s kind of definition of what we would think of. And so, designing for simplicity is really what we want. Simplicity is really what gives us the readability. Readability is a fantastic tool that will drive us in the correct direction. You’ve got to do other things for high-performance as well. But readability, I think, is one of the most profoundly important tools for all of those disciplines of managing complexity. It makes that code more modular, more cohesive, with better separation of concerns and all of those things. Those things make our code more readable. We can build little sentences up as we create that code. But even a non-technical programmer would be able to understand what was going on if you get the names right. So I think you’re right that readability is profoundly important. One of the real important tools in our toolbox.

Henry Suryawirawan: And also on the coupling, right? The other source of complexity. So you mentioned just now, if you write microservice, but you cannot independently release it or independently test it, this is also some pitfalls that people do. They write microservice, but before release, they need to coordinate multiple services working together and do the regression. So I learned from somewhere as well that coupling is one of the killer of agility. So if you cannot manage complexity, basically, you cannot have this agility. Thanks for emphasizing this concept.

[00:51:29] Other Modern Software Engineering Tools

Henry Suryawirawan: So maybe if we have some time to cover the tools. The tools that you mentioned can be the modern software engineering tools. What are those, actually? So you mentioned about TDD, Continuous Delivery. Maybe there are some other things that we haven’t learned from this conversation so far.

Dave Farley: Yeah. So there are a few other ideas. One of them is, again, going back to related to the DORA report. If we optimize our development approach to be able to work in smaller steps, and get faster feedback on our changes, we get to a more definitive position. We can make a small change, look at the effect of that change and understand it. Technically, I think one of the reasons why that is important is because that’s one way of limiting the variables. If we can make a small change and evaluate that change, essentially, in its own context, without worrying about the whole, the rest of the system, that’s going to give us a clearer picture of whether that change is good or not. As long as our system is modular and loosely coupled and all of those other things.

So that’s a good way of working. In Continuous Delivery terms, speed is probably one of the most profound tools that I use in trying to help big organizations to transition to working with Continuous Delivery. You just start saying, okay, so where are you now? You’re releasing once a year. So let’s try and move. What would it take for us to releasing once every six months? Let’s do that. Then every three months, then every month, then every two weeks, then every week. And just keep finding what the problems are that get in the way. You use the speed of feedback releasability to drive down and remove work and complexity from the development process until you can release every day. That’s a great way of doing things. It’s a great way of thinking about and structuring and exercising, changing the way that people in organizations work.

Deployability is another of these fantastic tools, and talking about microservices kind of points to that very clearly. If we want to be able to take an engineering stance, then really what we’d like to be able to do is to be as definitive as we possibly can about our evaluation of our software before we release. So I’d like to evaluate my software and have as much confidence as I can achieve before I release it. Not just put any old rubbish into production and wait for it to fall over. I’d like to be able to evaluate it in as close to the circumstances as I can imagine for it to be deployable.

There are two strategies, as far as I can see, that make sense to be able to do that. Both of them are driven by the deployability for software, which is why I think deployability is a useful tool. So one of them is that we could just decide that we’re going to treat our whole system as one thing. We’ll evaluate it altogether. If all of our tests pass, we’ll say, yes, it’s good and we’ll put it into production. Or we could decompose it into many independent parts, evaluate this part and say, this part is good, and now that can go into production. The second one only works as long as those things are independently deployable. If at the end of my deployment pipeline, I say this part is good, but now I’m going to test it with all of these other parts before I release it. Then the real pipeline is that second one where I’m evaluating everything together. To be able to do continuously, you have to be able to do that once per day.

And there’s a lot of complexity now because I’ve got all these separate repos and separate deployment pipelines and lots of inefficiencies with that way of working. Microservices are independently deployable. So I think you can use the deployability of a system as a tool. So we can say, we want to release changes to production. What is a unit of deployment? What is the thing that I can evaluate to the point where I’m happy for it to go into production without doing any more work? What’s that unit? That tells me the scope of evaluation in my deployment pipeline, and probably the scope of what goes into my repository. That’s a useful tool. It’s a definitive statement on how you structure your code in repositories and things like that. The scoping evaluation is a deployable unit.

All of these things, test-driven development, speed, releasability. All of these things are tools that we can use to amplify our ability to learn and our ability to manage the complexity of the systems that we build. So they kind of come together and that way of working amplifies those software engineering properties that we’re striving to achieve.

Henry Suryawirawan: I like the way you mentioned about the scope of the pipeline. So in your book, you mentioned that if your pipeline says it’s good, then it’s good to go. There’s nothing else that you should wait. No approval, no coordination with other teams. I think that’s also another key insights for me when I read the book is because sometimes when we have a pipeline, okay, all tests run successful and all that, but we still wait either for approval or awaiting other system to release first or something like that. But the concept of good pipeline is that when it’s good, you just release it. There’s nothing else that you should worry about.

Dave Farley: That’s one of those semantic diffusions that we were talking about earlier. I am the person that came up with the idea of the deployment pipeline. So that’s absolutely what I meant is that it’s deployable at the end of the pipeline. I tend to work in more complicated industries these days. So regulated industries, building medical devices, or finance systems or machinery. The challenge there is, “Oh, yes, but we’ve got to get this signed off by these compliance people.” “No, you’ve got to automate that as well.”

Tesla made a change to the Model 3 production line. This is a change to the factory. They upgrade the charging for the Model 3 car from 200 kilowatts to 250 kilowatts. That was a software change which reconfigured the factory to build cars differently, and it took three hours.

Henry Suryawirawan: Wow.

Dave Farley: So they put this software change in, it went through the deployment pipeline and then the factory is producing the new version of the Model 3 car. Space X are changing software on their rockets 43 minutes before the rocket launches.

Henry Suryawirawan: Wow.

Dave Farley: So you can do this. You can do this in even very difficult circumstances. But it does take a different mindset. What I would think of as an engineering mindset to do it.

Henry Suryawirawan: So all software developers who listen here, if you think doing Continuous Delivery is hard for your software, think about SpaceX and Tesla. How they can do it with cars and factory and rockets. So I think we can all aspire to become much better and try to build better software faster.

[00:57:42] 3 Tech Lead Wisdom

Henry Suryawirawan: So Dave, thank you so much for sharing your insights. All these conversations, I think, are really great. But unfortunately, due to time, we have to wrap up pretty soon. But I would like to ask you one last question before I let you go. So this is something what I called three technical leadership wisdom. Something like advice from you to all of us here, listeners, to learn from your journey or experience or from your past projects. So what will be Dave’s three technical leadership wisdom?

Dave Farley: It’s a good question. I think my first one, I think the industry as a whole, would be significantly better if we all started out assuming that our own ideas were wrong, rather than assuming that they were right. So go in thinking this is bound to be wrong, but I wonder how it’s wrong. I think that’s an engineering mindset. That sets us up for looking for the ways in which our engineering mindset is wrong. Thinking about the ways in which our software and our systems and our working practices can go wrong. I think that’s a much better way of organizing things.

If I could flip a switch and change the way that software development worked, I would make it so that everybody from now on that learns how to program, did it in the context of test-driven development first. Their first day of writing their first program, they started with a test. That’s the way that I would teach test programming. Because I think that if we learned it like that, there’ll be none of these arguments about whether it was better or not, because we’d know that it is. So I think that’s the next one.

The last one is I feel privileged to have found a career that I have reveled in. I have found consistently even on the difficult days, even when I’m struggling, I love what I’ve done for a living. It’s fantastic. I think that we should sometimes stop and think about that and think about the value that we bring to the world. We are the engines of change in our society. More than almost any other group of people. That means that we have the privilege of being in that position to do amazing things like tools and social media and the internet and all of that kind of stuff, which is fantastic on many fronts. But it also means we have a responsibility to do that in a thoughtful way and worry about some of the implications of some of these things. So I would also ask people to be a little thoughtful about the context in which their work is placed. That’s not somebody else’s responsibility. The Volvo emissions scandal a few years ago showed that saying that my boss told me to do this is not an excuse. The person who did it still went to jail. So we should be taking responsibility for our own work and thinking about it in a social context, as well as in a technical context.

Henry Suryawirawan: Wow. That’s really beautiful. I was guessing that you probably will say something about Continuous Delivery, but apparently it’s something else. So, Dave, I’m sure that people know where to look for you, but for completeness sake, if people want to connect with you and continue this conversation, where they can find you online?

Dave Farley: You can get me at Twitter. My Twitter handle is @DaveFarley77. Go to YouTube and search for Continuous Delivery and you will find my channel. There’s some good stuff there, even though I say so myself. My books are available through the usual sources. You can buy all of my books on Amazon. One of them, at least on Leanpub. So take a look for those as well. I have a blog site as well, davefarley.net.

Henry Suryawirawan: So in preparation of this conversation, I must say that your book is really insightful and thank you for writing that. I find it really good if we all engineers want to upskill and maybe go back to the basics a little bit. So thinking about learning and also managing complexity. So thank you so much, Dave, for your time today. It was a pleasure.

Dave Farley: Thank you. It’s a lot of fun. It’s nice talking to you.

– End –