#120 - Software Architecture: From Fundamentals to the Hard Parts - Neal Ford

[00:01:01] Episode Introduction

Henry Suryawirawan: Hello again, my listeners. Welcome to the Tech Lead Journal podcast, the show where you can learn about technical leadership and excellence from my conversations with great thought leaders in the tech industry. If this is your first time listening to Tech Lead Journal, don’t forget to subscribe and follow the show on your podcast app and on LinkedIn, Twitter, and Instagram. And to support my journey creating this podcast, subscribe as a patron at techleadjournal.dev/patron.

My guest for today’s episode is Neal Ford. Neal is the Director and software architect at ThoughtWorks. In this episode, we discussed all things about software architecture covering his three most recent books: “Fundamentals of Software Architecture”, “Software Architecture: The Hard Parts”, and “Building Evolutionary Architectures”.

We first discussed the definition of software architecture and how it relates to software design. Neal then described the two important laws of software architecture related to trade-offs and the why. Neal then explained why software architecture is difficult and discussed the hard parts, such as finding the least-worst combination trade-offs, understanding the importance of data, and managing coupling. Towards the end, Neal shared about the evolutionary architecture concept and some of its principles to support making incremental changes across multiple software architectural dimensions.

This is a jam packed episode on software architecture. And I really enjoyed my conversation with Neal. If you are a software practitioner, I highly recommend reading all the three books covered in this episode, as there are so many other insights about software architecture that we couldn’t cover in this conversation.

As always, if you find this episode useful, please help share it with your friends and colleagues, so they can also benefit from listening to this episode. Don’t forget to give this podcast a 5-star rating and review on Apple Podcasts and Spotify. It will help me a lot to make this podcast easily discovered by other listeners. Before we continue to the conversation with Neal, let’s hear some words from our sponsors.

[00:03:53] Introduction

Henry Suryawirawan: Hey, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I’m so excited. I have someone who I admire. Very well-known author. He was also one of the ex-colleagues that I have last time in ThoughtWorks. As we all know, some ThoughtWorkers actually produce a lot of books. And I’m very pleased to have Neal Ford in the show today. He has written a number of books, mainly around architecture. He’s actually the Director and software architect at ThoughtWorks for a long time. So, Neal, I’m really excited to have you in the show, looking forward for this chat.

Neal Ford: Thanks for having me, Henry. I’m happy to be here.

[00:04:25] Career Journey

Henry Suryawirawan: Yeah. So Neal, let’s start by knowing more about yourself. So if you can tell us your career journey, maybe mentioning your highlights or any turning points in your career.

Neal Ford: Sure. I’ve actually not had that many jobs, surprisingly enough. When I was going to university, I was dabbling in a number of different majors. My first major was actually journalism, which I think is where all the writing comes from. But then that was too subjective for me, so I switched over to engineering. My two-year degree was actually toward a mechanical engineering degree, but then when I started the advanced work for that, I didn’t like it very much. And so, I was paying my own way through university. So I had the luxury of taking some time out and figuring out what I really wanted to do. I had taken a few software courses, programming courses, and I love that. And I thought, well, that’s probably what I should do. So I went back to university and got my degree in computer science.

And then started on one of my two jobs in the software world, because the first company I went to work for was a consulting and training company called the DSW Group, and I eventually became the CTO there, and worked there for almost 12 years. And then I joined ThoughtWorks and I’ve been at ThoughtWorks for 17 years now. So, I don’t change job positions much over time.

But I’ve been increasingly doing software architecture before I came to ThoughtWorks and when I got to ThoughtWorks. In fact, when I was hired at ThoughtWorks, I specifically said application architect on my offer letter. But then when I arrived at ThoughtWorks during that era, when I met somebody, they said, “Oh, who are you? What’s your role?” And I said, “I’m an architect”. And their immediate response was, “Oh, we don’t have architects at ThoughtWorks. We have tech leads.” And I said, “Oh, okay. I guess then I’m a tech lead here”. Cause it was a very strong anti architect sentiment at the time. Because at the time, it was justified, because architect had gathered this reputation of ivory tower. Kind of disconnected from the business and coding and too much hands off. In fact, that was considered sort of a badge of pride, that you didn’t do any coding as an architect. And, of course, that was not the ethos of ThoughtWorks then or now. But we’ve gradually embraced the idea of being a more effective architect.

When I joined ThoughtWorks, I’d already written three books. And I was also speaking a lot on the speaking circuit. There’s a conference series here in the US that travels around every weekend, and does like 25 conferences a year. So I was involved with that. So I was doing a lot of public speaking about technology stuff. And when I joined ThoughtWorks, I continued doing that, and I continued writing. Wrote books on a variety of subjects. But probably one of the biggest shifts that happened a few years ago, this has probably been eight or nine years ago now, was just as I’d wrapped up. I wrote a book called Functional Thinking, which is about functional programming. And I was really into functional programming and really evangelizing functional programming. But after I’d done that for a few years, I realized I had done as much as I was going to do to move the world toward that. And it was up to the world now to decide whether they wanted to do that or not. So I started really actively looking around for areas that I thought were underserved by books and conferences, etc.

I remember seeing a news story that talked about 10 most desirable jobs in the US. Software architect was like number three. Then I started looking at these lists from all over the world and software architect was, you know, on the top 10 on a lot of these lists. But then you look at the other things on the list, things like pharmacy assistant. Well, there’s a clear career path from not being a pharmacy assistant to being a pharmacy assistant, but there’s no career path like that for software architects. Every architect is this weird snowflake combination of experience and breadth and what’s happened to them.

On that same conference circuit that I was on, Mark Richards was also on that conference circuit, and he was also the one who did a lot of talking about software architecture topics. And so, I got together with him and we started sort of asking a question, what would a curriculum for software architecture look like if you were going to teach a class? Because he had just started becoming a full-time trainer. So we thought, okay, what would a three-day hands-on software architecture course contain? We ended up using that course to iterate on material until we refined it enough to get to the fundamentals of software architecture book. So over the last probably eight or nine years, I’ve been focusing a lot on software architecture, because it’s one of those roles that’s constantly changing, and it’s gotten much more complex recently, cause of the kinds of architectures we’re building.

Architects have thrust themselves into the middle of every conversation because of microservices, because the transactionality used to be a concern of the data folks, the database administrators, all those folks. But when you get to things like microservices, that becomes an architectural concern too. And it requires more collaboration and more knowledge. And so, along with my embrace of software architecture as a topic, it seems like the field itself has gotten much more complicated, which has been nice if you’re talking about it, because there’s a lot of interest in it.

Henry Suryawirawan: Thanks for sharing your journey. First of all, there are only two companies. That’s quite amazing in these modern days, where you have so many companies available and people are hopping from job to job. And secondly, it’s about how you ended up focusing on architectures. I’ve been there as well. So having the time when architects were like people don’t look at them, right? They think they are ivory tower architects. To now, which is in the modern cloud native kind of like workloads, people crave for architects, because yeah, they find difficulties in architecting good software. And especially microservices and data and all that. So I think a good architect now is kind of like valuable.

Neal Ford: Actually, let me interject one thing because I find the same phenomenon happening right now with enterprise architects. So I actually believe it’s time to reclaim the role of enterprise architect, cause they have the exact same reputation that other architects had 15 years ago of being too ivory tower and too disconnected. So I believe there’s a revolution coming on the rethinking the role of enterprise architecture and what it looks like in the 21st century. And I’m actually working with a few of our clients on rethinking what enterprise architecture would look like the completely different world we’re living in.

Henry Suryawirawan: Right. Thanks for that input. So I believe as well, architecture is very important, which brings us to the topic of this conversation. We’ll try to cover three of your most recent books. The first one is “Fundamentals of Software Architecture”, which you just mentioned you co-wrote it with Mark Richards. And then second one is “Software Architecture: The Hard Parts”, and later on, the “Building Evolutionary Architecture”, which is the new edition.

[00:10:48] Definition of Software Architecture

Henry Suryawirawan: Let’s start first with the fundamentals of software architecture. To many people actually, when they hear about architect, architecture, some might have different interpretation or definition. But in your view, having written this book, what is actually a software architecture?

Neal Ford: Well, we specifically declined to try to define it in our book, just like Martin Fowler declined to try to define it definitively in his very famous white paper called “Who Needs An Architect?”. Actually, Mark and I and another author, Raju Gandhi, are currently working on a Head First Software Architecture. And the Head First book series, you know, is almost a graphic novel, and we are actually wrestling much more directly with the how do you define architecture and what is the line between architecture design in that book. We sort of usefully dodged it in the fundamentals book because it’s such a tricky thing to try to define, because it’s so many things, and a single definition has a hard time capturing all of the facets and all the nuance. But here’s our most recent stab at it.

We can’t define software architecture, but we can define a software architecture decision. And a software architecture decision is one where all of the possibilities have serious trade-offs associated with, because the real essence of software architecture is trade-off analysis. For easy questions, they’re easy and you can answer them. But what we bumped into more often are things that don’t have easy answers, and there is no best solution here. So now you’ve got to find what the least-worst combination of these forces are. That’s one of the things that Mark and I talked about is trying to find the least-worst solution, not the best solution. Because very often, they’re just fundamental things that are in conflict with each other, and you always have to compromise and always have to do trade-offs. And so I think that’s the essence of software architecture. It really gets to the common definitions of stuff that’s hard to change later. Or why is it hard to change later? Because they’re significant trade-offs associated with that decision, which means that you trade off some capabilities for other capabilities, which is the essence of a kind of a software architecture decision. Ralph Johnson’s famous definition is, “It is the important stuff, whatever that is.” But again, it’s important because it has serious trade-off implications.

We view architecture design not as a binary, but a spectrum. You can kind of see on the spectrum where you are. So, one far end is something like the color choice, the user interface element, which is trivial to change. It’s a literally less than a second change to make that design change versus changing from a monolith to microservices, which is a month or year long effort. That’s kind of the opposite ends of the spectrum. And then everything else is sort of in the messy gray middle area between architecture design. But I think it’s important to note that regardless of how much it is an architecture decision, there should be collaboration between the developers and the DBAs and the database folks, and the people who actually build the software system. Because none of these roles can be successful in a vacuum.

But at the same token, there are some decisions that require more thought and more trade-off analysis and more responsibility. That’s part of what makes the role of software architect difficult is because it is the nexus of so many responsibilities and questions that need to be answered. Because a software architect can see process, issues, and opportunities, and engineering opportunities and issues, and DevOps. They touch so many things and have such a unique perspective that adds to the weight of many of their decisions, because they’re the only ones that are really have enough perspective to make a lot of these decisions.

[00:14:29] Software Architecture vs Design

Henry Suryawirawan: So you mentioned a couple of times already, like architecture and design, or maybe some people call it system design, are they the same? That’s the first thing.

Neal Ford: As far as I know, there’s no standard definition of system design, which is the problem with architecture, design, and all these things. I mean, design is a facet of architecture, because to do architecture, you have to design things. But design is way too broad a word as well. Because I mean, anything that’s created by a human is design. We’re running into the limitations of language here, trying to define something that is so multifaceted, because software architecture is about designing the structure of the system, the components and relationships and defining the architecture characteristics. But it’s also about leadership. It’s about merging technology decisions with business decisions. How do I take a business case and reify that in technology? How do I make current technology work on the given constraints that I have? Physical constraints versus cloud computing versus where do resources go? They’re often the tech lead as well. So, how do I manage conflicts in the team?

One of the common things you learn, particularly if you’re an architect at ThoughtWorks, is dealing with strong personalities with very strongly expressed differing opinions about technology directions you should have on your project. That’s leadership and negotiation more than it is about technology choices. So, I mean, all these things make up the role of software architect, and I’ve only scratched the surface of the things that you run into in a lot of organizations that fall into the heading of software architects. So I think you’ve never seen a good concise definition because it’s impossible to have. It’s like, give me a good, concise definition of art. Some things just defy simple definitions and software architecture seems to be one of those things.

Henry Suryawirawan: So thanks for putting the emphasis on soft skills and leadership as well. Because that is part of the architecture job, right? To influence people, to explain the rationale so that the business also understands why certain things are difficult? Why there’s a trade-off?

[00:16:24] Laws of Software Architecture

Henry Suryawirawan: Speaking about trade-off, right? You have this law of software architecture. There are two laws. If you can maybe briefly describe them and why do you come up with this law?

Neal Ford: So one of the things that Mark and I wanted to do as we were working on the fundamentals book, it’s worth mentioning too, and this may be of some interest to your listeners, that what Mark and I have figured out how to do is to iteratively design books. So you’re from ThoughtWorks, you’re familiar with iterative software development. Most people think that books are these giant waterfall projects where you have to go to a dimly lit room, and think deeply for months and years at a time, and write all this stuff down, and finally release this huge thing and this massive release. What Mark and I have realized is that you need feedback, and that makes the material better.

When we started teaching the fundamentals class, we started this feedback loop, and we made a commitment that every time we teach this class, we’re going to make one substantive improvement to it. So the next time we teach it, it’ll be one iteration better. We did that for several years and we realized, hey, we’ve refined this material and we could actually write it down in a book. That’s exactly what we did. One of the things that we were trying to do in that book as we were going through that iterative process is, okay, let’s keep an eye out for things that seem universally true. Things that we can’t find an exception to anywhere. And we started with a handful of them and ended up with just two that survived that process. Those are these two laws that we codified. We actually wanted more of them, cause it’s cooler if you have 10 laws. But we ended up just with two that seemed to work as holistically as these did.

The first one is that everything in software architecture is a trade-off. That is kind of the defining characteristic of software is it involves a trade-off of one kind or another. And we built a corollary to that law that if you think you found something that doesn’t have trade-offs, it probably just means you haven’t figured out what those trade-offs are yet. This is sort of an homage to the very famous Donald Knuth quote about premature optimization in particular. His quote was rules for optimization. Rule number one is don’t do it. And rule number two is don’t do it yet. So that’s sort of the corollary we have.

And then the second law that we codified was that the why is more important than the how. And the way we got to that was one of the things that we wanted to do is we were teaching these materials, make it hands-on. We borrowed this idea of an architecture kata, which is a little time boxed exercise that forces an architect to go through trade-off analysis and figure out where components live and that kind of stuff. We were having people do that live, and they were drawing these pictures on flip charts and sticky notes, etc. Mark and I, cause we were teaching this at conferences, Mark was actually teaching this live quite a bit. We thought, oh well, would it be really cool if we built up a library of these solutions to these kata problems, and we started snapping pictures of all these solutions. And after we had done that for a few classes, we realized that they were pretty much useless. Because you could see how a particular team had solved that problem, but you didn’t know why they chose this trade-off versus this one. And we didn’t force the teams to write any kind of ADR or justification because it was a three day exercise. You don’t have time to do that. And we realized that how something works is okay, but why it works is so much more important.

And that’s true as an architect when you come upon an existing system that you don’t have any background on. I can explain exactly how this architecture works, how messaging works, how dependencies work, and analyze all the technical bits and bobs of the architecture. But they’re going to be some things that’s like, I don’t know why they chose this trade-off versus this other one because I might have chosen a different path. That is the essence of what’s interesting in architecture is the rationale and the reason why you did something, which is one of the reasons we stress this idea of creating lightweight documentation to capture that “why” of when you do make these important decisions so that reasoning that you went through doesn’t just entropy into the universe.

Henry Suryawirawan: It’s very interesting that when you mentioned about this architectural kata and patterns that work for people, but actually they become useless, right? Because yeah, I also learned that architecture is all about context. So when you mentioned these two laws, right? Let me repeat one more time. The first law is everything in software architecture is a trade-off. And then the second law is why is more important than how. So yeah, we can all explain about a certain architecture, how things work, the structure of the code and things like that. But why certain things are built that way, I think is also something that is important to capture maybe in the architectural decision records or maybe some kind of other format.

[00:20:55] The Hard Parts of Sofware Architecture

Henry Suryawirawan: Let’s try to move on to the second part of the conversation, which is about your second book, “Software Architecture: The Hard Parts”. We covered it in the beginning, where people refer software architecture as things that don’t change so often, very hard to change, and also things that are important. What is the rationale of you writing this second book, the hard parts? And what do you mean by the hard parts?

Neal Ford: Well, the way that book came about was as we were writing the fundamentals book, we kept bumping into these, and we started digging into explaining some problems. We realized, wow, this is way too complicated for fundamentals book, cause we were getting way into the weeds. And so, we took all those problems and set them aside and said, okay, we’ll deal with those later.

Over time, this pile got bigger and bigger, and by the time we finished the fundamentals book, we had this giant pile of difficult problems that we had kind of set aside. At one point, Mark, actually, we were in conversation, sort of made a joking comment of, “Well, our next book should be called Architecture the Hard Parts, and we should just cover all those things”. Then we said, well, you know, actually that’s a pretty good idea, because what it forced us to do was look at that giant pile of problems and figure out why are you so difficult? We basically applied our second law. What’s making these sets of problems so difficult? And that was really the basis for “Software Architecture: The Hard Parts” is an analysis. Of course, the thing, the ultimate, the short answer to that question of what was making them so difficult is that all of them really required this kind of continual trade-off analysis.

Mark got in trouble a few years ago at a conference for saying that there are no best practices in software architecture. And he softened that a little bit to say that in structural design, there are no best practices. Because when you think of the concept of a best practice, which is sort of out of fashion right now, but the whole concept of best practice is literally that when this situation comes up, I can just turn my brain off and do this thing, because it’s the best practice. It brooks no compromise. If they’d called it a better practice, then at least you’d have some wiggle room. But no, it’s the best practice and so, turn your brain off and just always do this. But so many of these things in software architecture, you can never codify a generic rule like that.

A great example of this is I have this shared functionality in microservices. Should I couple to it between two services or should I duplicate it? Both options suck. But you can’t say, you should universally do one or the other, because every time you run into that situation, you have to look at, okay, what is it going to do? What are the pluses and minuses on either side? Because of the domain behavior or the architecture characteristics or the load. All these different facets go into that decision every time. And that’s what makes a lot of these problems in architecture so difficult is you cannot codify an always do this in this situation. You have to do this trade-off analysis.

So the entire second book, the examples are really about coupling in distributed architectures. And the whole first half is about static coupling, which is how things are wired together. And the whole second half is about dynamic coupling, which is about how things call one another. I actually realized yesterday and would’ve put this in the book. The static coupling is kind of the three-dimensional view of the world because it doesn’t change as all the dependencies and you know how things are wired together. Whereas the dynamic coupling is the fourth dimension because it happens in time, because things are calling each other in request response cycles and that kind of stuff. The book is driven by those examples, but what we’re really doing in the book is codified in the subtitle, which is modern trade-off analysis for distributed architectures. We’re showing architects how to do their own trade-off analysis, because that’s what you have to do. There are no best practices in this world.

So there’s a little bit of a rhetorical trick in our book. The first 14 chapters are all written in first or third person. But the last chapter is called “Build Your Own Trade-Off Analysis”. And it’s written in second person cause it is the call to action to the readers that, Hey, we’ve shown you now 14 chapters of examples of how to do trade-off analysis. Here’s the list of all the tools we use to do that codified. Because what we’re really doing is encouraging architects to build their own toolbox of trade-off techniques. Cause one of the things that we also say is it does you no good as an architect to get good at generic architecture because there is no such thing as a generic architecture. You said it before. It’s all about the context. What your job is to get really good at doing trade-off analysis in your architecture. So what you need is a toolbox of trade-off tools that you can use. And so, we give you some generic ones, but we encourage you to build your own toolkit, and so we show you how to use some of the things that we did to start building up your own toolkit. So the fundamentals book is really about this kind of broad overview of software architecture. The hard parts book is very much focused on how do you do trade-off analysis in modern architectures.

Henry Suryawirawan: Yeah. I find that one of the rarer skills in a good architect is to do this kind of a trade-off analysis, that works in the context of the teams they’re working, the systems that they’re working. I think you mentioned a couple of times there are no best practices. Also in the book, you mentioned the seminal paper or book written by Fred Brooks, “No Silver Bullet”. So this is something that is always referred when people are talking about architecture. So I think thanks for the plug.

People these days, we all love to chase the hype-driven development, be it whatever cloud, containers, Kubernetes, and all these best practices that we think will solve all the problems. But I find that over the time, things tend to change. One thing that is good at this time, maybe at certain times or certain context, is not good. Thanks for having a reminder that in architecture everything is a trade-off. There’s no best practice.

[00:26:42] Least-Worst Combination of Trade-offs

Henry Suryawirawan: Which brings me to the next question that I want to ask. Because you mentioned in the beginning, when there’s no best practice and there’s no best design, we should try to find the least-worst combination of trade-offs. So how can we actually do this? And what do you mean by the least-worst combination trade-offs? Because there are so many things in architecture. So how can we do that? Maybe any tips for us to do that?

Neal Ford: Well, part of it is just doing analysis and figuring out what’s really important in your organization. One of the things we encourage architects to do is an architecture characteristics analysis. Because if you go to your stakeholders and say, what do you want? Performance, elasticity, scalability, and show them a list of 50 things. They’re going to say, “Oh, I want all 50 of them”. But you can’t have that.

And so, what we try to do is take the technical trade-offs and translate them into language that business people understand so that they can prioritize based on their priorities, not our priorities. For example, we try not to talk about extensibility and adaptability. We talk about rapid change as a competitive advantage. Well, rapid change really translates into extensibility and maintainability and low coupling between ecosystems. And so, finding out what the real priorities of the business is within any given organization is really the key to aligning the decisions you make with what they’re doing.

You made the point too that there’s a lot of shiny stuff out there. You got to be careful not to embrace too much shiny stuff and instead focus on business value. We talk a lot about value and translating technical decisions into value, because as you get further and further into technology realm, you get more disjointed from what the business is actually doing and what they view as valuable. And I think it’s important to stay grounded for that as a software architect. Because the more you can keep aligned with what the business is wanting to do, then the more trust they’re going to develop in you and the easier it is to build out what they want if you’re modeling what they want more closely.

Henry Suryawirawan: So I find that sometimes we don’t actually ask the business what is important. We just know the features, right? They give us the list of features, the functional requirements, some people say, and the non-functional requirements or quality attributes. Yeah. We almost want all of them, right? But we don’t actually, discipline enough, to actually explain, okay, if you choose all these non-functional requirements, basically it means X, right? Either the cost, the effort, or the resources required to build that will be a lot. So thanks for reminding us again to always try to explain architecture in business context. Prioritization should also come from the business, not just from technical point of view.

Neal Ford: And I think it’s worth doing that kind of architecture characteristics analysis for existing greenfield, brownfield systems, not just brand new ones as well. One of the things that we often do when we start working mid-project for a client is to go back and do and codify those things, because it helps clarify your decision-making process if you know what your ultimate goal is. Because exactly when you reach one of those places where, okay, I’ve got to make a decision here that’s going to push against one of these things. And you know, it’s that one of those least-worst situations. Knowing what the ultimate goal for a business is and having a good understanding of that will often help you weigh the scale one way or the other, where otherwise it would just be a tossup.

[00:29:51] Importance of Data

Henry Suryawirawan: So speaking about the hard things in architecture, right? So you mentioned this in the book, one of the most difficult parts is actually the data. To some companies, many systems, in fact, the data is like the most important thing. And sometimes it’s really hard to handle data. Especially when you make changes to software, you migrate certain things from the software. Maybe if you can a little bit share with us what is so critical about handling data? And why it is always the most difficult part in architecture?

Neal Ford: Well, Mark and I have slowly realized over the last few years that how important data is in software architecture, particularly when you get to things like microservices. Because even distributed architectures from 10 years ago, when we first started doing a lot of event driven architectures with COM and CORBA and technologies like that, we mostly still relegated transactional behavior to the database, because we were still talking to one big relational database. But then, domain-driven design came along. Part of that suggestion was this idea of a bounded context, which means that now the data persistence is part of the bounded context. It needs to be isolated at the architecture level, which means now suddenly transactionality is an architecture problem, not just a data problem. And it complicates a lot of stuff in software architecture when you start having to manage transactional behavior at the architecture level, because it’s really hard to model.

There was a book written in 1993 called “What Every Developer Should Know About Object-Oriented Design”. Translated in modern terminology, one of his observations in that book was that that kind of transactionality is the worst kind of coupling you can have in any kind of architecture, because you’re forcing wider and wider parts of your architecture to unify on a single value. Basically, transactionality means you’re stopping the universe for a moment to align things. You can’t really do that in high-scale architectures, or at least not and still have high-scale architecture. So that’s definitely one of the problems that comes into microservices. But we’ve also noticed, over the last few years, a more awareness of these dual uses for data, operational data versus analytical data. One of the dysfunctions, I think, we’ve tried to do for a long time as architects is use a single mechanism for wildly different purposes. So we tried to create the relational database as the sole source of all this knowledge, but now we need to do all this reporting and analytics off of it. And there are all these things about transformations and all this other stuff.

The problem with that is, this goes back to the fundamental problem you run into in architecture is that your database schema is one of the most intense implementation details of your entire architecture. And the more you let implementation details kind of leak out into your architecture, the more brittle your architecture becomes. Meaning, you make one change somewhere and it breaks a whole bunch of stuff. Well, trying to do this kind of data warehouse, ingestion and all, you’re trying to do that at the database schema level, and it’s going to be brittle, and it’s going to break a lot, which is exactly what we discovered. And so that’s exactly why in the “Software Architecture: The Hard Parts” book, we added two co-authors, Pramod Sadalage, who’s our colleague at ThoughtWorks and wrote the “NoSQL Distilled” book with Martin Fowler and the “Database Refactoring” book with Scott Ambler. So he’s an expert in operational data and how to deal with those kinds of evolutionary database changes. And Zhamak Dehghani, who, of course, created this idea of data mesh.

One of the best ways I’ve seen it resolves a lot of the issues that we’ve had of reconciling analytical data and distributed architectures like microservices. It’s a nice clean solution, sort of based on the idea of service mesh, but a little more advanced. But I think a really encouraging thing, because what Zhamak has basically figured out is how to overlay two different behaviors on top of an architecture. Because we’re basically taking the operational architecture then overlaying the analytical part with very specific, well-defined connection points. And that’s possible to do with other kinds of crosscutting concerns as well.

All of those things make data a much more interesting topic and became one of the things that we highlighted in the hard parts book. Because that’s one of those things that, as a modern architect, you have to reconcile with how do you manage transactionality? How do you design your architecture around transactionality? Because it changes a lot of the trade-off decisions you have to make. So it’s something that needs to be in your decision toolkit.

Henry Suryawirawan: Yeah. Speaking about data mesh, I had Zhamak in a few episodes before, and I think it is quite a revolutionary concept these days. I think it tries to adopt things like service mesh concept, DDD concept, product thinking concept and things like that. So I think, yeah, dealing with data, especially the so called the division between operational data, analytical data is always hard. So I think data mesh offers some kind of a unique way of approaching this kind of problem, especially as you scale, right? It may not be in the beginning where you have probably like a smaller number of services or data, but I think as you grow in terms of organization, data mesh will be something that is quite interesting for you to adopt.

Neal Ford: We have a database chapter in our book, but it really just covers the architectural design principles of data mesh using the same terminology we used throughout our book. But what Zhamak realized when she wrote her book is that we were lucky in that we’re really just talking primarily to technologists, developers, and architects and data engineers, etc. But you start talking about analytical data and you start touching every part of the organization. So she has to have a much broader mandate, because everybody in the organization she’s got to have a story for it and explain it in terms that they understand and get. So it’s a much, much wider argument that she has to make because she touches so much of the organization. We’re a bit luckier that we can just stay in the technology realm, and it makes our case a lot easier.

Henry Suryawirawan: Yeah, that’s so true, right? When you speak about data, you first need to understand what the data represents. Any kind of business meaning, business context, what’s the importance of this data, the types, accuracy, and things like that. So yeah, definitely is something that is really hard, especially for some software engineers who just love to play with technology.

[00:35:54] Coupling and The 3 Co’s

Henry Suryawirawan: You mentioned a couple times about coupling, right? And it is probably one of the most fundamental things that you mentioned in the hard parts book. Maybe if you can give some tips about coupling. What should an architect or developers think about coupling? And how can we actually try to be conscious when we make a certain decision that will relate to coupling?

Neal Ford: Well, it’s hard to summarize, because we wrote 450 pages about this subject, and we’ve got another 200 pages that didn’t make it into the book. One of the insights that we had was coupling is too broad a term, and we actually split this out into static coupling versus dynamic coupling. It helps you think about that, cause it categorizes. Static coupling is how things are wired together. So it’s the dependencies for things.

Actually, back in the first edition of the Evolutionary Architectures book, we created this concept of an architecture quantum, which is sort of like a bounded context in domain-driven design, but expressed in architecture terms, because bounded context already has a definition in that realm. An architecture quantum is like the smallest functioning piece of your architecture that runs standalone. So, all of its data dependencies and everything is self-contained within it. But when you start thinking in those terms, and it’s also the scope for a set of architecture characteristics. It’s common in microservices that different services may have different sets of architecture characteristics, like different levels of performance or security. The quantum is really the scope for a set of architecture characteristics. So it’s really a way of analyzing kind of operational coupling for things.

And we talk about the static coupling of quanta, which is basically what does it takes to bootstrap this service, that is the static coupling of that service. Some companies, and we’ve actually participated in an exercise to do this, is going through their architecture and saying, okay, if this thing breaks, what else won’t function? That is the static coupling of that architecture. It’s all the dependencies between things in that architecture that need to function for that architecture to function. That’s static coupling, and we talk about how to analyze static coupling. There are a few metrics to look at that and etc.

But then dynamic coupling is, okay, now you’ve got these services up and running and functioning. What happens when you start calling from one to the other? And one of our insights in this book was we had all these examples where we ended up focusing on these three forces that just ended up starting with “co”: communication, consistency, and coordination. So communication is synchronous versus asynchronous. Consistency is, am I trying to make this workflow act like an atomic, a single unit of change, or is it an eventual consistency? And then coordination, am I using orchestration for my workflow or am I using choreography?

The reason we ended up on those three forces was when we had all those examples, we kept trying to figure out a way to analyze it so that if you just change synchronous to asynchronous, but it shuffles the entire trade-off chart. There was no isolated change. And at one point we realized the reason for that is that this is a three dimensional conjoined space, that every time you change one of those sliders, you have to reevaluate the trade-offs for the other two as well. In other words, you can’t change one of these things in isolation, because changing from synchronous to asynchronous communication, it turns out affects the trade-offs on more orchestration and choreography, which affects your consistency. And so, that’s part of the analysis piece that we built in the dynamic coupling is how to do trade-offs for these forces that are conjoined like this?

And that’s basically the finale of the book is transactional sagas. How do you manage transactional sagas? So in the “Microservice Patterns” book, Chris Richardson names two sagas, which are transactional workflows. We ended up naming eight different ones, which is all these three forces taken in every combination and analyzed the trade-offs for each one, because it turns out that all those combinations are either legitimate patterns or anti-patterns and are chosen or not chosen for a particular reason.

Henry Suryawirawan: Really insightful and very interesting things that you mentioned just now, right? The three forces that you have to look at when you change something, you will have to re-look at everything, the trade-offs, the context. Make sure to check the book. So there’s a transaction saga chapter. I saw that you list down all these different options. If you choose one thing, that there are certain things that you have to consider.

[00:40:11] Evolutionary Architecture

Henry Suryawirawan: Speaking about software architecture, things will change, right? When you decide something today, it may change maybe one month later, even one week later. So, which brings us to the next book that you’re writing, “Building Evolutionary Architectures”. And I know that you have the next edition, the second edition coming in December. So first of all, how can we evolve our architecture? I thought making architecture decision is really hard, and things should not change. Very hard to change. But how can we evolve our architecture? And what’s the new thing that you added in this second edition of the book?

Neal Ford: Yep. So there are two things to think about if you’re trying to evolve a software architecture. The mechanics of how you do that, and then the structure that allows you to do that. We basically split our book into two parts, and, actually, the second edition is much more explicit around that separation where we weren’t quite as firm in that separation in the first book.

The mechanical part of this is really a great insight from Rebecca Parsons, who is currently the ThoughtWorks CTO, but she joined ThoughtWorks as an architect. Before she was at ThoughtWorks, she worked in the evolutionary computing space and came up with this idea. This is a great example of a mashup. In fact, we were talking about Zhamak earlier. Zhamak is a great example of what happens when you take an expert on distributed architectures and plunk them down in front of an analytical data problem. What kind of insights might you get? And the insight we got there was data mesh. Well, this is exactly what happened with Rebecca. What happens when you take a PhD computer scientist who’s been working the evolutionary computing world and plunk them down into enterprise architecture space? She came up with this mashup of using this mechanism from the evolutionary computing world called fitness function as a way of thinking about objectively measuring software architecture characteristics. What she realized was that’s really the essence of evolving a piece of software. Because if you decide, okay, to be successful, this software has to have a certain level of performance. Well, if you build a fitness function that measures that performance every time you make a change to that software, then you’ll know that it is going to be fit for as long as you keep that fitness function running.

It’s sort of analogous to when you think about evolving our problem domain. So we can separate structural design into your domain and the architecture characteristics like performance and scale and all that sort of stuff. Think about your problem domain. How do you manage regressions in your problem domain as you make changes to it? Well, we have unit tests. We have functional tests. We have user acceptance testing. In fact, if you’re a big enough company, you have an entire division called QA that’s devoted to this idea of not letting things change in an undesirable way.

What we don’t have is a similar mechanism for software architecture characteristics, and that’s what fitness functions are. They’re the unit test equivalent, but for things like performance and scale and elasticity. But they can’t be as simple as unit tests. So a lot of these are performance monitors, etc. But it’s a way of identifying the really important success criteria for the architecture from a non-domain standpoint, and building protections around those things so that as you make changes to the system, either voluntarily or involuntarily, because, let’s say, we’re using this persistence framework and it goes away, and now we need to swap to another one. We don’t want to make that change, but we have to. By having those fitness functions in place, protect the important part of that library so that as we swap it out, we know that the new one is functioning as a suitable replacement and not giving us a bunch of nasty surprises. So that’s the mechanical part of evolutionary architecture, and that’s really the secret to how you evolve things.

It’s also really kind of a through line of figuring out what real engineering looks like for software systems. And I think the continuous integration idea of being able to make small changes and then verify those changes right away. Cause the problem with modern software systems is that they consist of a hundred thousand, a million, several million individual moving configurable parts, and you can change almost any one of those parts at any time. So how can you ever trust it if you can’t verify it? But we have mechanisms to verify it every time we make a change to it. And so, building that kind of verification ecosystem allows us to have a 4 million part software ecosystem and still make changes to it with confidence because we’ve wrapped really important characteristics and make sure that we haven’t broken those things as we make changes to it, as we evolve it, as we swap out one piece for another. And so, that’s really the essence of the mechanics of evolutionary architecture.

Henry Suryawirawan: Yeah. Fitness function is something that I also had a chance to study it during my university degree. Speaking about evolutionary computing, I did that as well to build some kind of machine learning kind of programming, genetic programming at that time. It’s something that is very interesting, because you focus on what do you want to achieve as a performance, so we adapt that to architecture. So it means that, okay, maybe we select certain architectural characteristics that are important. And every time you make the change, you evaluate whether these changes actually still fit in the terms of fitness function. It still fits for the purpose.

[00:45:26] Incremental Change

Henry Suryawirawan: And speaking about evolutionary architecture, you mentioned about three important things. Like it has to be a guided change. Guided maybe by the fitness function or the business change. And it has to be incremental in a multiple architectural dimensions. So how can we actually make incremental change? Because, again, coming back to architecture, I thought many people refer to it as the big things, not the smaller thing, like lines of code and things like that. How can we make incremental change to architecture? This is probably quite tricky for some people.

Neal Ford: Depends on how well you’ve structured your architecture. That’s one of the advantages of microservices, because they’re so decoupled from one another. I can make changes to one microservice and only test it and not worry about that change rippling out to the other. So that allows me to make smaller and smaller units of change.

But one of the great examples that we gave in, actually, both additions of the book, but it’s even more highlighted in the second edition, is this case study that came from the GitHub engineering blog. And it’s an example of making changes in architecture even when you are doing very aggressive engineering thing. So GitHub does continuous deployment. They average 60 deploys a day. So they have a deployment pipeline that runs through this gauntlet of tests. There’s a blog entry in the GitHub engineering blog called “Move Fast and Fix Things”. And it talks about them developing this little tool called Scientist, which allows you to conduct experiments. It’s basically a feature toggle framework. If I want to replace some discreet functionality, you can put it in place and it’ll run both the old and every once in a while, run the new and compare the results, so that you can chase out any bugs and other stuff. All happening behind the scenes. All while doing continuous deployment and all this other stuff. They’re literally making an incremental change. Changing the way they merge two files at GitHub. Because they used to show out to a shell script and do it. Now they wrote their own internal version of it. And so, they changed out that discreet functionality within their architecture while doing 60 deploys a day without generating a single bug or a single breakage, because they were making that change. That is the essence of being able to make an incremental change within an architecture without breaking any of the surrounding stuff around it, because they built a testing framework around that.

That’s actually a category that we refer to as a fidelity fitness function. Because one of the things that we constantly find people needing to do on the big scale, they need to do this with a mainframe, is I really need to replace this functionality with this other functionality, but I’m not a hundred percent sure how the old one really works, because it’s COBOL code that’s been around for 50 years and nobody can dig. But I really needed to produce the same result as the old one, because markets are based on the results of this that we don’t actually understand how it produces those results. You can build the same kind of mechanism where, okay, let’s call the old one, but now gradually build the new one and call and compare them on a constant basis every time we make a call, and gradually come down and replicating that behavior. Even from a domain standpoint, you can implement that kind of incremental change as long as you have the ecosystem support behind it. Because, ultimately, that idea of Scientist is really just a specialized version of an architecture fitness function. This fidelity fitness that says, okay, I need to replace this and make sure it has the same value. So building an ecosystem around being able to do that allows you to do that.

Henry Suryawirawan: So that’s why it’s also important to build the supporting structure, maybe the infrastructure, the tools, the continuous delivery, the kind of like feature toggles. All these tests must be in place so that you can try to have evolving architecture. It’s not something that we just decide and, you know, hey, over the time we just focus on features, but actually also we have to think about all these different aspects of architecture.

Neal Ford: Well, going back to my engineering metaphor from before, you couldn’t really build a rocket ship before you’d invented calculus. And you can’t go from, well, I know how to do addition and subtraction. I want to build a rocket ship. It’s like, well, you’ve got to learn some calculus before you do that. So I mean, this is really the essence of engineering and software is this kind of ability to automate change and verify different pieces in ways that give you confidence that you can move forward aggressively.

[00:49:32] Principles of Evolutionary Architecture

Henry Suryawirawan: So in your book, you also mentioned a few principles of evolutionary architectures. Maybe if you can high level give us some guidance. What should be the principles if you want to aim for this evolving architecture?

Neal Ford: Well, certainly isolation, decoupling gives you the essence of being able to build an architecture that allows you to change. One of the things that we really talk about in the second edition is this idea that really originated before domain-driven design, but is highlighted by domain-driven design, this idea of not allowing implementation details to leak. The further they leak out of a context, the more brittle it makes your architecture.

I’m a consultant, so I get to see a lot of interesting sort of train wrecks. We have a client who has basically opened up the table schemas into their integration architecture, where anybody in their organization can just hit any database table anytime. And they’re shocked that their architecture breaks all the time. Cause you make a schema change in one table somewhere, and it breaks like six systems that were just doing queries against this. That’s insane! You’re letting this implementation detail leak out in a way that really damages your architecture. And so, part of this principle is isolation. It’s really this idea of bounded context and keeping implementation details as tight as possible, and then managing contracts between those things.

We spend a lot of time in our book talking about the contracts between pieces of software. How things call one another? And how much information’s there? How much information’s not there? Because that has a big impact on the way you can evolve the communication parts of your software system. Looser contracts give you more flexibility to evolve their interaction versus really tight contracts, which introduce more brittleness. There’s a big focus on architectural brittleness in the second edition of the book, which we view as the most damaging thing.

Henry Suryawirawan: So yeah, one of the interesting things you mentioned in the book is also about Postel’s Law, right? Be conservative in what you do, but be liberal in what you accept from others, in terms of communication between two different services. So I think all these things are really important for architect to know about. Hopefully, one day all of us can aspire to build a more evolving architecture, where we have all these fitness functions running every time we make a change to our software.

Thanks, Neal, for your time. I think it’s been a really insightful conversation. I really learned a lot about architecture. Especially, in preparation of this conversation, because I kind of like skimmed three of your books, and tried to digest everything at once. So there are still so many insights in those three books. For those of you who haven’t read the books, make sure to check it out because I think it’s really insightful if you are into the architecture thing.

[00:52:01] 3 Tech Lead Wisdom

Henry Suryawirawan: So, Neal, I have one last question before I let you go, which is something that I call three technical leadership wisdom. So think of it like advice that you want to give to the listeners here, maybe from your lessons learned or experience, maybe in software architecture or software in general. So what will be your three technical leadership wisdom?

Neal Ford: I think the best advice I can give architects, especially aspiring architects. One is nobody is ever completely fleshed out in this role. And so you need to view this as a sort of a bag of techniques that you build up over time. So you mentioned earlier, the ability to do a compelling presentation in front of the business. A presentation skills should be part of your bag of tricks as a software architect. Being able to craft a really good convincing email should be in your bag of tricks as a software architect. Conflict resolution. But no architect comes into this role, and in fact, no architect has every single trick they need. But you should view this as, “Oh, I’m building new skills”.

For a while at ThoughtWorks, we had this sponsorship program where senior people would sponsor junior people. We had this one really undesirable client with a lot of bureaucracy and a lot of other stuff that most people hated. One of my sponsees was there. So I called him at one point to get in touch with him, and I said, “So, how are things going there?” And he said, “Oh, I love it at this client”. And I said, “Really? Most people really hate it there”. And he said, “No, I love it here because it gives me a great chance to work on my consulting skills”. So he viewed that as, “Oh, this is a challenge that I can meet by working on this part of my toolkit. Hey, I need to work on my consulting skills”. So no architect is ever complete. In fact, impostor syndrome is a big problem with architects, because you don’t think you’re qualified enough. Turns out you are, but nobody has all the qualifications. So think of it as adding to your portfolio of skills all the time.

The second thing I would say is that, and we touched on this as well, that the way you get to be a software architect is by demonstrating technical expertise. But to be successful as an architect, you need to have a lot of soft skills. You need to have conflict resolution and leadership and being able to conduct a meeting that doesn’t spiral out of control and keep on an agenda and there’s a host of those things. As you keep moving to architecture, that’s the part of your toolkit that you should really focus on first, are some of those gaps that you have. Cause you’ve been filling in the technical gaps cause you enjoy that. Some of the things you don’t enjoy are conflict resolution and leadership, but some of those things are necessary to be a good success as an architect.

The third thing, I would say, and this is one of the things we point out in our fundamentals book, is that the thing that gets you to be a software architect is technical expertise. You’ve demonstrated a great technical expertise. The problem with that expertise, though, in the software world is that any level of expertise you have has to be maintained, and there’s an overhead for that. And actually as a software architect, so that expertise is your technical depth, but your breadth are all the things that you’ve been exposed to that you’re aware of, but not necessarily an expert in. The breadth is actually more important for a software architect than depth. Because it’s better for me as an architect to know that there are five different concurrency solutions to this problem than to know how to solve this concurrency problem deeply in one of those solutions. Cause as an architect, you need options, because you’re doing trade-offs, and you need to be able to trade-off one thing versus another. And this is another one of those painful things for tech leads who have become architects, is you start to surrender some of your deep expertise in favor of a more broad perspective on things. And people hate to surrender expertise, but trust me, it is for the better long term, the more breadth you can have.

So one of the things that I recommend that people do is gain breadth as quickly as you can. So if you’ve always been on a Java project, see if you can go and park yourself and pair with an architect on a .NET project for a month or six weeks, or a data science project, or a migration project. Because the more exposure you get to different things, the more you’re adding to that breadth and the more tools you can add to your toolkit, which ultimately, I think, makes you successful as a software architect.

Henry Suryawirawan: Wow. I really love your insights. Maybe I’ll call it software architecture wisdom, especially the last one. Surrender your expertise. I think some of us, especially techies, where we have invested our time and effort in specific technologies, and as time changes, things also change, right, in terms of technology. So you have to surrender your expertise and try to build the breadth, not just the depth. I think that’s really insightful for me.

So, Neal, for people who love this conversation, they want to reach out to you, talk more about maybe architecture or ask you about specific things. Is there a place where they can reach out to you online?

Neal Ford: Absolutely! I have a website at my name, NealFord.com. I was early enough on the internet. I was able to get my name as a vanity site, so NEALFORD.com. And I’m also on Twitter for as long as it’s going to last. It looks kind of dire for Twitter right now. N-E-A-L and the number four D, which is the kind of little version name, the shortened version of my name, available in both those places.

Henry Suryawirawan: So thanks again for your time, Neal. Really a pleasant conversation and I learn a lot. Thanks again for your time.

Neal Ford: And thanks for having me. It was a terrific time, and it was a great conversation.

– End –