#187 - The Tao of Microservices - Richard Rodger

[00:01:30] Introduction

Henry Suryawirawan: Hey guys, welcome back to another new episode of the Tech Lead Journal podcast. Today I have with me an author. An interesting book, I must say. The Tao of Microservices. His name is Richard Roger. Richard, welcome to the show.

Richard Rodger: Henry, thanks for having me on. Yeah. Let’s start arguing about microservices. I’m looking forward to it

Henry Suryawirawan: Right.

Richard Rodger: You know, everybody agrees on what microservices are and how to do them. There’s no controversy at all.

[00:01:55] Career Journey

Henry Suryawirawan: I’m sure we will get to a lot of different perspectives about microservices and monolith and things like that. But maybe let’s start the show by asking about yourself first, if you can introduce, maybe, telling us any highlights or turning points in your career that we all can learn from you.

Richard Rodger: Yeah, I graduated in 97. and I studied mathematics and philosophy. And by random luck, the mathematics course had a C++ module. So I learned to code on VI, original VI, on VT100 green screen terminals in the attic of the maths department. So, of course, I ended up in going into web development, right? It was just the pre-dotcom boom era. The website that I’m most proud of, the fastest one I ever built was in C, pure C. It was a real estate website. I didn’t know about malloc, so everything was statically allocated, which is why it was really fast. But literally, every operation was a buffer overflow waiting to happen. But it remains the fastest website I’ve ever built.

I spent the first 10 years of my career as a back office Java developer. I pretty much went to no conferences, did almost no community stuff. I did read a lot of stuff, but I never really kind of engaged outside of what I was working on until I discovered Node.js. and the Node. js community and kind of got out of my shell.

I got into startups and the first successful startup that I was involved in that did exit. After I left, unfortunately, but built the core tech. I was the CTO. We built something like Heroku for mobile apps, but it used JavaScript on the backend. Used the Rhino engine for JavaScript, cause we wanted it to be easy and quick to, you know, kind of like, almost like no code nearly, quick to build a backend for mobile apps.

And developed a love for JavaScript, which is a weird language, but I don’t know, once you kind of get over all the strange stuff, yeah, I don’t know. It just kind of clicked in my brain, you know. I like the object prototype approach rather than sort of hard classes. When I was doing Java, I read the Gang of Four Design Patterns book. I could tell you off by heart, all the patterns, right? Visitor, everything. And then when I discovered JavaScript, I realized you didn’t have to actually use any of the patterns. You can just go pretty much functional.

So myself and three other guys set up a consultancy based around Node.js. And, you know, sometimes you can get lucky. People talk about product market fit. And if you have to ask the question, do you have product market fit? You don’t. Now we weren’t a product company. We tried, but we were just a consultancy. But Node was taking off around that era, it’s sort of 2012. And we were pretty much the biggest Node consultancy in Europe. We started a few conferences. I started speaking at various conferences, that type of stuff, and the whole thing took off. And within the space of two or three years, we had a big business, which is kind of crazy so that was super fun, super stressful.

And then we discovered that JavaScript is not a great language for writing large enterprise systems. But our clients wanted us, cause it was the hype. So that’s how we ended up using microservices or the idea of microservices. But also ended up with a kind of a weird take, which we can get to in a little bit. But basically the limitations, I mean, it’s weird, isn’t it right? I love the language JavaScript, but the limitations of JavaScript, and that’s, I mean, I code in TypeScript now. But you’ve got to remember JavaScript back then, 10 years ago, was pretty bad, right? It was kind of a sucky language. Once you got any sort of code volume, you can’t even refactor it. I mean, there’s no type of safety. Hopefully, you wrote good unit tests, but they’re probably tied to the implementation. So refactoring is almost impossible. So out of the badness of JavaScript, we ended up using microservices to compensate.

It’s funny how things like that go in software, isn’t it? You have this idea that there’s some sort of perfection or ideal way to build systems, but it’s very, as the physicists say, it’s very path dependent.

So where do I get to? I have to get to today because you asked about the career history. So yeah, consulting is great, but, if anybody’s worked in consultancy, they know it’s really hard work, tons of travel. And I’d still wanted to do a proper SaaS, a proper traditional VC funded SaaS. So because I’ve been doing a lot of conference speaking, I thought I’d spot an opportunity in the event tech market. Specifically a SaaS solution to help people run booths and do sales lead capture and recruiting and… because I mean I was speaking at like a couple of conferences a month. But organizing it all using a spreadsheet which kind of sucked.

So I sold out of the consultancy and sold up to the other partners and set up a SaaS business. Did the angel round, took in a good size round, built the product, everything was fantastic until March 2020. So we lost all our customers to COVID in the space of a month, and we have to pivot. Eventually though, it took us about a year to figure out what to pivot into, but we realized that a key part of the market that we’d been looking at was developer relations. And luckily, developer relations still has to happen, even if there are no conferences. So yeah, we pivoted into that as a market, which is where we are now.

However, it’s never a simple straight line. Because of COVID, we never did a series A. And you know, how do you survive? So guess what? Consulting. So I tried, I tried to get away from consulting, ended up doing it again. Luckily, we’re kind of refocusing on product now, but I ended up doing consulting again for a few years which, yeah, really put the ideas around microservices to the test again, because I guess the interesting thing was that, although we didn’t make the entire event tech platform open source, about 80 percent of it was open source, plugin based. We’ll get into why that approach was taken, but you know, that means that if you’ve built something that’s based on a microservices architecture with open source components for one vertical, event technology, it was a good litmus test for the adaptability of microservices. Can they be used in other consulting projects, in other domains? And I think we found that the answer was yes. But for a very particular reason, I’ll just tease it, but we’ll get into that later.

Henry Suryawirawan: Right.

Richard Rodger: And that bringss us up to today. So yeah. The one regret I have, and if anyone here is, if any of your listeners are a junior engineer, I would say that a couple of years in a much larger big tech company is probably worth doing at some point in your career.

I nearly, nearly worked for SAP when I lived in Germany, but decided to chose a startup instead. But I probably should have taken the job at SAP for a couple of years. I mean, I, I’ve ended up having clients that are big companies, so I suppose I’ve ended up understanding them that way. But even if you do intend to do a startup, I think it’s important to work at a really big company to understand how they work so that you can sell to them. So that’s I guess one regret. But apart from that, I’ve had lots of fun. I’m still coding. Coding for 27 years. Not gonna stop.

Henry Suryawirawan: Wow.

Richard Rodger: Do not intend to stop. Oh, that’s the other thing. Yeah, you’ve got 20 good years from about 18 to 38. After that you’ve got to compensate with good design, because the old nighters don’t work anymore and you can’t code yourself out of complexity just with brute force because yeah, you lose IQ points. The brain starts degrading, unfortunately. So, yeah, enjoy it, enjoy it while you can.

Henry Suryawirawan: Thank you for sharing your story. I think it’s really interesting. Sounds like a roller coaster as well, right? So you’ve been into so many different journeys.

[00:10:12] The Tao of Microservices

Henry Suryawirawan: So definitely, there are a lot of learnings, but today we’ll focus more on the microservices, right? In which you have written this book called the Tao of Microservices. But first of all, I’m quite interested, like, why do you choose this title, you know, the Tao? Is there any specific meaning behind it?

Richard Rodger: Yeah, yeah. It’s interesting when you write stuff. Sometimes you can sort of forget to fully use a metaphor. And I’m currently doing the second edition of the book, so hopefully I’ll get to actually use the metaphor fully this time around. What it refers to is the fact that what I saw a lot of people doing was just assuming that microservices were mini web servers. And yes, okay, microservices is a very broad term, like everything in tech. And sure, a mini web server with a small API and its own database is a microservice. But it’s horrible architecture.

You know, all those articles that you read criticizing microservices or people saying, oh, we’re going back to a monolith, because we had a really bad experience. They are all legitimate criticisms. I came at microservices from a totally different place. So I felt there was a different way. Tao was kind of the way. I felt there was a different, more philosophical way to approach it. Let’s talk about that because I think that’s where a lot of the potential has been missed. It doesn’t mean it can’t still be taken advantage of, because the way we develop software is continuously evolving, right?

So I worked as a researcher for a while. I had my very first startup failed. It’s very important to have a failed first startup. I was completely broke, so I ended up working as a researcher at a local university and ended up working in the telecom space on IP telephony, which involves something called the Session Initiation Protocol, SIP, which is kind of like HTTP except state based and has loads more verbs. And if you imagine two phones, right, one phone ringing another, then it’s, you’re sending like ringing state messages. I don’t know, Henry, have you ever played around with any IPs? Yeah, it’s horribly complicated.

Henry Suryawirawan: No.

Richard Rodger: I wouldn’t recommend it. In the startup that failed, I’d really gotten into, if you go back to the Gang of Four design patterns, really got into the command pattern, right, where you kind of encapsulate commands in the system and you try to express all your business logic as commands, right? Now, I mean, this is a very, very early version of something like CQRS or event based approaches, event sourcing, perhaps. But very locked into the code, very hard coded. My experience working with IP telephony led me to think about the power of pattern matching. So the problem with receiving different state of messages for IP telephony is that you have to decide what to do with them. So you’re trying to write a little state machine, but first you have to match what’s going on. And the application that we were building, you have to provide customization.

So we just ended up writing a really, really simple pattern matching algorithm. And so, I mean, if you’ve written something, if you’ve used Haskell or something like that, right, you understand this idea of pattern matching. In fact, I think it’s being introduced in a whole bunch of languages now. I think there’s even something for JavaScript. It’s literally the most brain dead, simple approach thing you can think of where you’re just looking at a bag of properties and you’re just matching against some of them. I mean, you can get more complex and you can have regular expressions or matching interior properties or whatever, but it doesn’t really matter. All that matters is that you have this bag of properties, which might be a JSON document or a HTTP request, and you’re just picking a few that you care about, and then using that to do more business logic.

So when we got around to building Node.js applications, I kind of put these two ideas together and said, how did we turn the command pattern into a component model that uses pattern matching? So here’s the thing about consulting, right? Have you ever worked as a consultant? Have you ever, like, built websites for people?

Henry Suryawirawan: Yes.

Richard Rodger: Yeah, okay. So here’s the problem. It’s the same job every time, really, okay? Because what we do as consultants is we turn Excel into administration interfaces, right? And there’s always user management and permissions and projects and groups. And there’s always data ingestion. There’s always report generation. Maybe some of them have a mapping application or something like that, but business applications are always the same. So if you’re building a consultancy, you want to be able to reuse stuff, because that makes you more efficient, better job. And Node was very open source, so a lot of the npm registry was growing exponentially. I know that has its own problems, right?

The problem is a lot of those components were technical libraries. So stuff for like sending email or reading an Excel file or database drivers or ORMs, right? That type of thing. Yeah, they’re components, but the problem is every one of them has like an API that you have to learn. And they’re infrastructure. They’re not business logic. So let’s say you’re building a referral system. Your users can invite each other into the application. The logic is pretty much the same every time. So how do you build a referral component that you can reuse? That was the problem that we were facing.

So we built the system where it was event based. The only interface between components was a JSON document. And the routing was based on pattern matching. Your system consists of… This is before microservices, right? It’s a single process application. Just a single node process running on a bare metal server, right? It’s just the way we built things back in the day. I built an entire newspaper system using this subscription model, Apple Pay, everything. But the whole architecture of the system consisted of very isolated components. The only interface input and output was effectively JSON documents. The routing to decide where a message went was based on pattern matching.

And that meant that you could define a referral service, let’s say, by a set of messages so an invitation has happened, or generate an invitation code, or give this user a reward based on the fact that the invitation code has been redeemed by defining JSON documents. And you could have schemas if you wanted, right? But essentially you didn’t have to. So when a message comes into the system, you might have some tags that say, this is a referral message and it goes to the referral component.

So that was fun until about 2014 until, like I said, things get too big, right? It’s very hard to manage a system like that. It’s hard to scale it. I mean, you can for sure, you can deploy multiple monoliths and put them beyond load balancers. All that sort of stuff. But then we started noticing things like serverless and Docker and microservices idea was kind of in the air. And it was just an obvious next step to say, okay, well, why don’t we take the referral component and turn it into a microservice. And then we can deploy it independently and we can scale it independently and get all those benefits. But the important thing is it’s not a mini web server. It’s not a mini API endpoint. It only knows how to receive and send JSON documents. That’s it.

So that’s how we ended up with that. That’s how we ended up in a place where we were using microservices, but with a component oriented approach. So I think the way to do microservices, the Tao of microservices is not to be obsessed with the network and Kubernetes and all that other stuff. To think of them first as software components. Okay. And all the other stuff is just a consequence.

Henry Suryawirawan: Right. I think this is a quite unique perspective, because I think from all the journeys of microservice that I have followed, right. I think it has gone from ups and downs. For example, from monolith to microservice, and then it goes back into monolith these days. Some people call it modular monolith. Or whatever that is. But I think that there are still places where microservices could actually work. And sometimes, some people advocate things like domain-driven design, bounded context to actually create a proper boundary.

[00:18:22] 3 Core Technical Principles

Henry Suryawirawan: But the one that you just mentioned, is actually to think of microservices in terms of three different things that I could gather from the book, right? The first is about designing the message first so things like the JSON document you mentioned. Second is the pattern matching, which is kind of like, you know, associate the kind of shape of the message, and then you route it differently to different services. And the last thing is about additivity, you know, like, think of it like you can compose, for example, if you want to work on some workflows, right, you compose that kind of workflow, using different components or microservices.

So maybe, yeah, this is, I think, the true, maybe the Tao of the way of you defining microservices. So tell us about why these three are so important, and why do you advocate this approach instead of, you know, some of the more popular way of designing microservice?

Richard Rodger: Yeah, it comes back to this idea of components first. You actually mentioned the most important one which is composition. If you’re designing an object oriented system these days, you you you should prefer composition over inheritance, right? So inheritance. People realize that inherent…, you know, basic inheritance is probably a bad idea. You end up tying yourself in knots, whereas composing things together is the way to go. So, I mean, what’s a component system? It’s Lego, right? It’s meant to be like you’re 10 years old and you were building a spaceship with Lego and the things just click together. So how do you get to that point? How does that work? Because everybody who’s worked on large enterprise systems know that they’re not Lego, that refactoring is horrible. I mean, everybody’s had a iteration where you have to stop the world and refactor. So the composition element is really, really important.

And I think the pattern matching gives that to you with a really nice mental model. So first of all, I should say separately, the implementation details here don’t matter. We built an open source framework for doing this, but it’s actually pretty small. You can do this in any language. You could literally write one from scratch for a greenfield project or whatever. You don’t particularly need to use a framework. It’s more of a philosophy than a framework. Pattern matching gives you composition, because it allows you to easily go from the general to the specific.

So one of my go to examples is, let’s say, user management. When you start building the system, you might only have, you might only decide there’s one type of user in the system. And there’s a little bit of logic to deal with that. So I don’t know, assigning a user to a project is one JSON document with a few tags that you can pattern match on. But that only gets you through maybe two or three iterations, because of course systems don’t have one type of user. They have admin users, and they have project managers, and they have root permissions and all sorts of crazy stuff. So if you have an assigned user to project message, that is general. Very simple, right? It’s just adding an entry to a database table and setting up very basic permissions. That’s your first implementation.

But you might have additional business logic that has to happen if it’s an admin user. There’s a special value that has to go in or something into the database. Or they have to be put into a central registry. There’s some sort of extra thing that has to happen. So in most systems, what you’d end up doing is adding a whole bunch of if statements in that logic. So if it’s an admin user, do this extra stuff. But that’s not composable. What you want to do is you want to leave the old code alone, pretty much, that just handles normal users. Run that code and then also run extra code that handles the admin case. So what you can do is either before or after.

And again, this is pretty simple to implement, because all you’re doing is you’re saying, take a look at JSON documents and if a few of the properties match, do something. So I can say, okay, my properties match for adding a user to a project, add the user of the project. Oh, there’s also a tag that says this user is admin, so I’m also going to trigger the admin user message. You mentioned additivity, right? So that’s additive. The old code stays the same, and then here’s the new code, and you click it into the system like a Lego brick. And you can do that for all sorts of things.

It’s particularly cool for cross cutting concerns. So one of my other favorite examples there is, let’s say you want to have like a creation time and an updated time in your database rows across the board for all of your data entities. Well, if you encode the data entity operations, you know, saving and loading as messages. Then the pattern matching can intercept those messages and add in creation time and save time, auditing, permissions checks, whatever, all sorts of fun stuff without that being visible in the business logic code.

The whole point is you want to keep your business logic as business logic, right? Business logic shouldn’t be setting creation times or worrying about permissions or any of that sort of stuff. So if you have these additive components, you can see how it’s pretty easy to start saying, well, maybe they actually live on separate parts of the network. And maybe the messages can flow over the network to make these things happen. Because it’s all just pattern matched routing. The business logic code has no knowledge whether something is on the network or not.

So you can run the whole thing as a monolith, and this is how we develop, right? We run the whole thing as a monolith locally. I mean, local development with real microservices, and I have done this and I have, you know, killed my machine with like 30 Docker services and crazy stuff like that. These are not lessons that weren’t learned without a lot of blood. But you can run locally as a modular monolith, because you’ve got your modules. And then our favorite deployment these days is split up the messages by domain and deploy them as lambdas. And that’s really cool because you don’t have to worry about running things in a separate service locally. You can use Terraform or something like that to automate the deployment of the lambdas. So you, you don’t have You’re not worrying about working with lambdas and deploying stuff to debug and all that. Lambda development is kind of sucky.

And then the other big thing that the component based approach gives you is really simple mocking. So anybody who’s ever tried to build unit tests and build mocks is from, well, I don’t know, have you heard of the banana monkey jungle problem?

Henry Suryawirawan: No, I haven’t.

Richard Rodger: So this is a, this is a… I tried to imagine it in a very posh English accent, because I learned about this from a friend of mine who used to be a fusion scientist. He worked on the joint European Taurus. So he’s a very clever chap. So he must be right.

So the problem with a lot of, and I mean this is a big problem in Java. The problem with writing unit tests in a lot of systems is that you have all this dependency and you have to do dependency injection, all that sort of crazy stuff.

So you need a banana to unit test the banana, right? But unfortunately, to get the banana, you need a monkey. And then to get a monkey, you have to get the whole jungle, right? So you basically have to bootstrap your entire system to do a small unit test. Now if the only way that your system is exposed is via JSON documents, then mocking them is really simple. You just hard code the response. You don’t need all the rest of the infrastructure. You don’t need to have mocking utilities that will build complex APIs.

And I think this is, it’s a point I make in the book actually I just remembered. To get Legos, to get components, you can’t have complex interfaces. You think about, modern languages. They try to present the class as a component. But classes have methods, they have interfaces. They have static methods. They have member variables. They have annotations. Are they singletons? Are they… and it only gets worse, right?

Because in order for the language to have power, the class construct has to have lots of different ways to interface with it. So now you have to learn a new API each time. And someone has to model that whole idea of user types in a system that would have to be modelled. So now you’ve got an interface with it somehow, and it’s very hard to compose together.

I was very inspired by Unix pipes. So if you think about it, that’s one of the most successful component models ever built in the history of computing. And that’s because the interface is really simple. It’s just a stream of bytes from one process to another. But that has its own problems. It’s a little bit too simple. I don’t recommend building, I mean, people have, right, but I don’t recommend building enterprise systems with streams of bytes. So you need to just up the complexity a tiny little bit more, right? So JSON documents.

I used to be really big into XML, by the way. I’ve written XML parsers and all sorts of fun stuff. It’s cool, but it’s so painful. Which is why I like JSON so much, right? It does make life a little bit easier, even though it could really do with having comments, but anyway.

If you up the complexity, just a tiny bit, and you allow the messages that go between components to be JSON documents, and then you, you know, apply that leniency principle where Postel’s law, I think it’s called, where you’re lenient in what you accept and strict in what you omit. It’s just powerful enough to build enterprise systems with when you use pattern matching. It covers about 95 percent of use cases. For the 5 percent it doesn’t, you always have to be able to expose the actual underlying API, right? Always have a get out of jail free card. But 95 percent is pretty good. I’ve kind of covered a lot of ground, but I’m trying to get the philosophy in one body.

[00:27:55] Messages First

Henry Suryawirawan: So I have some follow up questions, actually. So when we think about these messages, right, you keep mentioning about JSON documents and things like that. Some people also these days have event-driven architecture, right? Even like modeling software as events. Is this message you are talking about the same as kind of like an event? From my understanding, it seems not. I mean, like it’s not fully 100%. And then how does this pattern matching work against the message? Because when you talk about composability, I think it’s kind of like easy to, how should I say, easy to think about when you talk about function calls or maybe like Unix pipe. But when you have different microservices and you have different network calls and databases, how does this composability work? And not to mention the transaction as well. So maybe if you can maybe elaborate a little bit more on these two parts?

Richard Rodger: Yeah. So my concept of messages is a bit more general. And I’ll actually refer to the book, because I did spend a bit of time in the book, creating an architectural structure for thinking about it. So event sourcing is very much a unidirectional flow and that’s, it’s great idea. And it’s this core benefit is that writes go in one direction and reads come from a different place. It’s a kind of a one way system. The real world is a little bit messier, especially since you might be dealing with legacy systems and all that sort of stuff. And a great many APIs are designed to be request response. So I identified kind of two axes that messages could operate on.

So this is, one of them is whether they’re synchronous or asynchronous. So synchronous messages expect a response. You send the message and you expect a response. You’re waiting for a response. So if I save something to the database, I expect a response confirming that it’s saved, that gives me a new ID, something like that. Asynchronous messages are ones where you just emit them. You don’t know, right? Somebody might act on them. You just don’t know. And then the other one is whether they’re consumed or not. So you can think of a classic message queue where you’re sitting there as a listener and the message comes on the queue and you take it and process it, and then the message is deleted. Amazon Simple Queue Service, something like that, right? Or you can think of something like Kafka where there might be multiple readers.

So one of the people that I found really inspiring in the early days of microservices, and if you go onto YouTube and look for his talks, they’re absolutely fantastic, is a guy called Fred George, who built a lot of the early microservices systems. One of the examples he gives is insurance quotes. So you’re trying to do dynamic insurance quoting. So somebody types in their details and they hit give me a quote and the system can process for about 30 seconds. So the way he implemented that system was that messages would go out to lots of different microservices that could handle different types of quotes. And then they would all provide quotes and then there would be some business logic to rank them or something like that.

That’s an example of a system where the messages are not, they’re not consumed by a single service, they’re broadcast. So when you’re building a system, you can think of it in terms of whether the message is a request response or whether it’s more like event sourcing. So whatever way you get messages from microservice A to microservice B, whatever way the routing works, whatever way the pattern matching works, give yourself that freedom, because you will need it. But I mean, sometimes the criticism that I get and something that sometimes I find people struggle with is that, they think I’m handwaving when I’m saying, oh, message writing is do whatever you like. So let me give you a very concrete example.

So recently we built an AI chatbot for podcasts, right? It lets you kind of interrogate guests. It’s kind of fun, just as a kind of fun side project. And that runs in Amazon, uses Simple Queue Service. And if you think about it, there’s all sorts of stuff. You have to download the RSS feed, then you have to download the audio, and then you have to get transcripts, and then you have to chunk the transcript, and then you have to put it in the vector database and do all the fun stuff. Talk to the models to get the vectors. So it’s very much a batch processing flow. It’s very much messages go from A to B through different Lambdas. And if you built Lambdas and use SQS, the way that works is you have named topics. And one Lambda will submit a message to a named topic. And then another lambda is listening to the named topic. So that doesn’t fit my model, because if you think about it, those named topics are hard coded. But I want to have a situation where I’m free and easy and I’m not stuck with topics, right? I just want to send my message and it magically gets to the right place. So in my modular monolith that I’m building locally when I’m developing the system, I just say that my, let’s say my audio downloader microservice will listen for messages that say, here’s the RSS. And then it gets the RSS and it can look for the audio file download, which means that my component that downloads the RSS can emit that message. Everything happens in one process. I have a tiny little algorithm that looks at the keys of a JSON object in memory. And then makes a method call to the right place. So somehow I have to translate that to Lambda SQS architecture.

Initially, when we started doing it, stuff like that was moved into configuration. So you would say, okay, these patterns of messages, it’s like when you have a hammer, everything is a nail, right? These patterns of messages all map to this topic here. So I have an extra bit of code that or an extra plugin, let’s say that I deploy when I’m deploying to Amazon that intercepts the messages. This is a common pattern, right? Intercepting it. It’s just pattern matching like anything else, except this time it doesn’t do any business logic. This time it does transport. It’ll transport the messages for you. And for a long time, that configuration was done like any configuration system, right? It was in the code. You configure that and they’re option configurations for these plugins. But one of the reasons that I decided to do a second edition of the book is model driven architecture. So this is giving anybody who’s my age nightmares at this stage.

So about 20 years ago, it was really, there was this, a lot of hype around UML diagrams and you would literally architect the system as a blueprint. And then all of the junior coders would just type it out, basically, right? Total failure, because that’s not how software actually works. But I think there was a kernel of an idea there that was really good, which is that, and sorry, then you can combine that with like domain-driven design and things like that to say there has to be a higher level abstract description of the system that’s above the code in some way. and I mean, we, we use that all the time these days, right? So YAML definitions for serverless deployments, Kubernetes, infrastructure as code, right? But you could have architecture as code as well. And not diagrams, right? Not visual programming, because that never works. But you can essentially use all of the very robust configuration tools that we have these days.

There’s a lot of them, right? So take the ideas in Terraform or Pulumi or things like that and create a type-safe model of the system where you are listing your database tables and you’re listing your services and you’re listing your messages. And then you can do things like say, oh, here’s the Lambda deployment context.

And in that context, these messages go to this topic in SQS. And that’s now a deployment configuration, completely separate from business logic. So I never have to, when I’m writing my podcast subscription code, my audio download code, I never have to care. I just don’t care how they talk to each other. I just know. I can take for granted that the message will get delivered. And then you add in lots of the modern observability, like Open Telemetry for actually debugging that. And yeah, it’s kind of fun. We’ve had fun these days.

[00:35:55] Pattern Matching

Henry Suryawirawan: Right. I can also see a few alternatives to pattern matching. I mean, tell me if this is not correct, right? So you can use something like a server path routing, right? So where you have HTTP slash A goes to A service, slash B to B service. Or you can also take the message bus approach, right? Where you have some business rules, so to speak, right? In the message bus to actually route to different topics. Or maybe in like RabbitMQ, you have different channel and exchanges and things like that. Is that also the same kind of thing when you mentioned about pattern matching?

Richard Rodger: Yeah, I have a very big hammer. They’re all nails. So the issue is the power that you get from keeping everything to pattern matching is that things like unit testing and mocking get much easier. Or especially if you have junior developers on the team, the only API that they interact with is, essentially, you can send messages or receive them. Now, we do add a little ORM layer, right? An object relational mapper that actually becomes messages anyway. Talk about that in a minute, cause that is quite powerful.

But when the junior developer is writing business logic code, all they should ever be doing is sending and receiving messages. Loading and saving stuff to the database. That’s it. So if those messages are coming in from RabbitMQ or they’re coming in from HTTP request or whatever, the routing should translate that first. So for example, on the HTTP side of things, what we’ve written for ourselves is little gateway plugins that will translate. So locally, I’m old school, we still use the Node library Express for local development. So we have a little gateway that can translate Express inbound HTTP requests into a JSON document.

Yeah, there are, you know, a subset of requests where you need to be able to access the headers and all that sort of stuff to do auth. But again, you can use the pattern routing, because that’s a cross cutting concern. You can put it elsewhere. And you can just restrict business logic to saying, here’s a JSON document. It happened to come in this case from HTTP. But you don’t know, maybe it’s a unit test. Right, so you know the way unit testing APIs is really painful, because you have to simulate HTTP requests.

You know, there’s a few people that do it really well, like Fastify. Again, I’m obviously using Node.js examples here. Fastify is a really good system for simulating inbound HTTP requests. But the issue there is to use that you have to commit to Fastify. The problem is if you’re deploying to Lambdas, well, now you’re not really using Fastify. I mean, you could, but generally you just want to use the events that Amazon sends. So now you have a whole different system. So you have another little plugin that can translate Amazon events.

We recently had a scenario where we built a system on AWS. Spent six months building the system, enterprise system. To be told one month before deployment that corporate policy was that everything had to run on Azure. So you can imagine that in most scenarios that that’s a total disaster. If you have used any Amazon APIs directly, you’re in trouble. But because we’ve modeled the whole system as messages, all we have to write, and we already had some because we obviously built other Azure systems, all we have to do is write things like a little gateway plugin for Azure functions, which have a slightly different inbound message form. Which by the way is a JSON document anyway, right?

There’s another good analogy for this approach, which is AWS EventBridge. So if you’ve used EventBridge at all, it’s actually very similar. Or Erlang/OTP, similar style. These are all either influences or same basic idea. So yeah, it is a very, very big hammer. And I’m not advocating for a very specific approach to building large scale enterprise systems. But at the same time, it’s all driven by this idea that you really want a composable component model. And the only way to get there is to have a simplified API. So you always want to hide things like RabbitMQ or the fact that it’s HTTP request at the end of the day.

One final thing I’ll say on that is very much aware of the seven fallacies of network architecture. And I know very well that what I’m advocating for is to pretend the network doesn’t exist. But that’s a different discussion.

Henry Suryawirawan: Yeah, I think the one that you mentioned, like it’s called transport independence in your book, So yeah, even though you use web server, HTTP web server, or you use message queue, or you use some other mechanism, I think it doesn’t matter as long as so called translate that into messages, and you use some kind of pattern matching to be able to route to different services.

And I think this key about composability is something that maybe not many developers naturally think that way, because if you think about adding user types, most of the people will just refactor the code and tweak the portion and do if else and that kind of thing. But I think doing pattern matching maybe is like the functional paradigm as well, right? The functional programming paradigm.

Richard Rodger: Oh, 100%. I mean, I stole the idea from Erlang. Totally.

Henry Suryawirawan: So I think these are definitely the heuristics that you advocate whenever you design microservices. And obviously in your book you also covered about model driven, you know, like modeling code…

Richard Rodger: Yeah.

Henry Suryawirawan: … which, yeah, some people like it, some don’t, but we probably won’t discuss that.

Richard Rodger: That’s the secret. It’s all code.

[00:41:18] Monolith vs Microservices

Henry Suryawirawan: Yeah. So maybe as part of this everlasting debate about microservice versus monolith. What’s your take about monolith, by the way? So do you actually ban monolith at all in your software design or do you actually find that monoliths still have a case?

Richard Rodger: No. So, you know, because we use a modular monolith effectively for local development, it’s a wonderful get out of jail card, because that project that we’re setting about where we had to move to Azure, we converted everything from Lambdas to Azure Functions, set everything up. And in the last week, it turned out that the admin who could give us appropriate permissions was on holiday. So there was no way for us to get enough permissions to make the system work and go live by the deadline. So our solution ended up being to run a VM and run the system as a monolith in production. And now there were some performance issues, right? Because this is where microservices are great for scaling, because parts of the system can scale independently.

So we did have to deal with some performance issues. But we had a get out of jail card and that we could, it was nothing to do with software architecture, but the dynamics of the politics, the project meant that the only way we could go live was to deploy a monolith. So we still have that option. And I would not have had that option five years ago. I mean this is, when people say microservices are crazy and monoliths are easy, they’re right. And when people say, you know, if you’re building an MVP, maybe you should just build a monolith to begin with. And I actually agree, right, just deploy it on a bare metal, have a server or something for 10 a month.

But wouldn’t it be nice if it was a component based model that you could turn into Lambdas or a big Kubernetes deployment without having to rewrite the whole thing? It doesn’t have to be my specific approach to pattern matching. It doesn’t have to be Node. js. You just have to start from a place of saying that my system should be composed of components and all the components should have the same. Just like Legos, they should have the same interface. And that’s really where all the power comes from. Yeah, it’s one of those silly debates that isn’t real.

But it comes from places like, I mean, I’ll tell you, we were brought in to a large enterprise client. This would be maybe 2016. Microservices were getting really popular and they built a whole load of mini web servers and they were running about 60 microservices. And the problem is that even though they’ve given all the developers really powerful laptops, as a developer, you couldn’t run the system locally. So every developer was also given a large Amazon instance to run the system so they could develop against it. And I mean, of course, if you come into a setup like that, where you have a team of 50 developers that can’t even run the system locally, of course, you’re going to go, well, microservices are utter nonsense. Why on earth would you do this to yourself?

The only people that I’ve ever seen that have actually managed to make the mini web service thing work is Netflix. I mean, they had the resources and the brain power to make it work. If you look at their open source and all their tooling and all that sort of stuff, wow! I mean, it’s amazing. I mean, yeah, there’s a fantastic way to control everything. But it kind of tells a story, doesn’t it? In that they needed all that brain power and resources to make it work? You know, and I, as an advocate of microservices, you know, surely the easy road is to use Netflix’s stuff, but I don’t have a team of 20 Stanford graduates to make it all work.

Henry Suryawirawan: Not too mention running all those Netflix tools, right? I think that might require a set of people, right?

Richard Rodger: I mean, I’m sure there are other people who’ve done a really good job of it. Netflix open source, obviously, so you can kind of tell. Yeah, it’s, what’s the Spider Man quote? With great power comes great responsibility. So microservices are really cool and you can do really cool stuff, but as the Australians say, they can be a massive foot gun as well, right?

[00:45:17] Network Fallacy

Henry Suryawirawan: So maybe let’s go into what you mentioned just now about the network fallacy and also the data. I think these two typically are the two biggest things, right? So network is, you can’t assume network is always reliable and fast. And second thing is about data, right? When you have your data passing across multiple services, there’s no atomicity anymore, right?

Everything is like an eventual consistency. And how do you reconcile if there are failures in one part of the service into the other?

Richard Rodger: Yes, yeah. Okay, so here’s the thing. Just because the network is unreliable doesn’t mean a monolith is reliable either. You know, when I used to deploy monoliths in C and Java, stuff like that, VMs have run out of disk space, because the logs have used up everything, right? Or memory leaks, right?

I mean, when I used to run Java servers, we used to just restart them every 24 hours, because of memory leaks, right? But the problem is if you just randomly restart a server at 4 AM in the morning, well, that’s not 4 AM for everybody. You could be in the middle of a transaction. And it can just die. So if you walk into a supermarket, do you think that the prices are 100 percent accurate across every single label of every single product? No.

There’s actually, uh, in the US, I think the law is, the federal regulation is that they allow a 2 percent error rate. So the price that you see on the shelf is incorrect up to 2 percent of the time. I think, and again, this is inspired by Erlang, you have to design your systems to accept a baseline error rate of some kind, because weird stuff is always going to happen. Yes, the network is going to make that a bit higher. Of course, the trade off. It’s always a trade off is that you get scale. So you have to look at your business case and think about what the mitigations are going to be. If you’re building a consumer app, you know, that’s to do with user generated content, I mean, how many times have you used Reddit and it doesn’t load, it’s like retry, load again, right? So, I mean, who cares? If it was a network failure, just reload.

I’ve built banking apps, and I can tell you that, they’re very careful with their mainframe COBOL systems that do the actual transactions, but I can also tell you that there are backroom staff that reconcile problems. I mean has your bank been 100 percent correct over your entire lifetime? You hear these stories of people who have like magically have 10,000 appear in their account, right? This type of stuff. So it’s a fallacy or telcos and their five nines uptime. I mean, have you ever had a cell phone call fail? It’s a fallacy to assume that you can build an error free system. So then you have to look at the business case around what the mitigation is going to be. And is that, is it serious enough that you have to have a backroom staff overnight, like banks fixing things? Do you solve it economically by compensating your users in some way?

Or you’re Google where your profit margins are so great that, you know what, if your search results aren’t that good anymore, it doesn’t really matter, right? So, yes, the network, I mean, you know, the seven network fallacies are 100 percent true. But you don’t deal with them by trying to make the network infallible. You deal with them by accepting that the system overall, even if it’s monolith, has a baseline error rate and that’s a business requirements issue. And sometimes it just, I mean, for a lot of systems, it just doesn’t matter. It’s just hit reload in the browser and keep going. Especially a lot of back office enterprise systems, that’s just the way they work. It’s a little bit different for some types of consumer apps, where a baseline error rate might mean that you get hundreds of support requests a day, right? So that’s going to suck.

So here’s, I mean, here’s the interesting thing that the modular component based approach can do for you. If you’re finding that there’s a microservice A and a microservice B that are really co-dependent. And you’re getting an error rate due to network issues that’s too high. Well, combine them into one deployment architect, right? Turn those network calls into back into function calls. Now you’re still modular because the interface is still just the messages. But you may find that your beautifully designed partitioned domain-driven architecture has 10 percent of cases that need to be optimized. And when you have to optimize code, the code just looks bad, right? So you stick them together and deploy them as one artifact. So that’s the networking side of things.

[00:49:30] Handling Data Consistency

Richard Rodger: The data side of things is interesting. So first of all, it’s kind of similar in that if you actually look at the details of transactions in traditional database systems, if you actually look at what the guarantees they offer, they’re not actually that good. If you look at the specific details, like if you look at Oracle, right, there’s actually, I think four, there’s four different transaction levels. You might see old data. And you might end up having to use them, because you need the throughput in your system or it might not matter. So designing a system where you need to do distributed transactions. Yeah, that kind of sucks. Avoid it, if possible.

Unfortunately, it may not be possible, especially if it’s a large enterprise system with multiple teams. But hey, you know what, even in those scenarios, there’s usually like a lot of different monoliths, so you still have the same problem, right? So then you’re in enterprise service buses and reversing things and all that sort of stuff. You just use SAP, right? SAP solved this in the 1980s. No, seriously. Like if you look at R/3, if you look at the SAP system, it’s like the perfect microservice deployment system. They have solved all this stuff. It’s just, you need like $2,000 a day consultants to actually make it work. So especially if you’re not building mega scale systems, try very hard to keep transactions contained inside one service, one unit of deployment. And if you’ve got this modular architecture, then again, you can stick things together.

The other thing which I advocate in the book is don’t just go to transactions automatically. There’s loads of other approaches you could use. I mean, even sort of brute force things like having a batch process that will correct data that’s gone slightly wrong. Reservations. So reservations are really cool. That’s kind of a standard algorithm to prevent like double spending. I mean, I won’t go into the specifics of it, but you’re basically creating database entries that basically block other things from happening. But it doesn’t need, it just needs atomic writes. You don’t need transactions. I’m a little bit naughty when it comes to the database, I have to say. I don’t mind microservices sharing databases or tables. I don’t know where that idea came from. This kind of obsession with the purity of the database per microservice.

Henry Suryawirawan: I guess if I may provide my understanding, right, it’s because of the leaky internal data model that if you use two different services sharing the same database, right, the other service could just easily, oh, I see this table, I see this data, I’ll just query it. And, you know, think of it as if, like, it’s my data, right? But actually, yeah…

Richard Rodger: Yes, yes. Yeah. So the data becomes a secondary interface, which now needs to be managed. I mean, I don’t see how monoliths are immune to that problem either, right? Okay, so that kind of leads me on to the way that we use this idea to represent data. So I don’t know if you were aware of another argument going on in parallel between people who dislike ORMs and people who speak ORMs, you should have ORMs. And I’ve used both, obviously. My original C code was directly talking to MySQL. And I’ve used Hibernate, sharded Hibernate for Java, which was, wow. Wow. Heavy going.

Okay, so here’s the thing. I think one of the big learnings from the NoSQL movement was that joins are overused. And almost always, 90 percent of cases, you really don’t need to join tables. Now that does introduce certain inefficiencies for sure, but the nice thing about that is it keeps your business logic relatively simple, because you have very strong ideas of specific entities, like there’s a user and there’s a shopping cart and there’s a line item. And if you denormalize, again, you have to add in maybe batch processes to correct things and reservations to keep everything consistent. But mostly if you denormalize and you don’t use joins, mostly you’re okay for most enterprise. And certainly if you’re doing a consumer application for scale, you’re going to need to denormalize anyway. Leave the joins to the reporting engines and things like that, right? So I mean, we preferentially use things like DynamoDB these days. And you can’t really do joins on that, right? Yeah.

So one of the underlying things I find that where people are not enthusiastic about microservices is because they also tend to have to give up the idea that you can join tables and write custom SQL inside your application. Microservice component based approaches tend to push you in that direction. That’s a big thing to give up. But I think it also goes back to the idea of being inspired by Unix pipes, because the simpler that you can make the shared interfaces, the less trouble they give you. So even if you have shared tables, shared schemas, if people aren’t doing joins, they’re mostly okay. And then if you only expose the data via messages anyway, if the schema drift or something like that, then you can add interception messages that will temporarily adjust until you can figure things out, right?

So in the book, I give some examples of migration strategies where you deploy multiple services or translator services to handle those types of scenarios. You know, they pay us good money cause this stuff ain’t easy. So I don’t think there’s any easy answers to that. I just think it’s not as, it’s all about trade offs, right? So my personal trade off is I don’t really use joins apart from reporting. I’m still here. I am alive.

Henry Suryawirawan: I think the point where you make, you use less joins. So I think in your book, you mentioned about data duplication, right? So maybe think of, like instead of normalizing and doing joins, right, you can also duplicate data. Yes, it may stale a little bit, but also like thinking in terms of eventual consistency, you won’t get like a strong consistency anyway, right? So I think that’s really, really interesting view as well. So I think we can’t cover everything interesting in your book.

Richard Rodger: You could, you could tell I keep going for another three hours. Yeah.

Henry Suryawirawan: So definitely for people who are interested in this new way of thinking, you know, the Tao of Microservices, I do recommend checking out Richard’s book. I think it really provides you with a unique different views of how to tackle microservice problem.

[00:55:47] 2 Tech Lead Wisdom

Henry Suryawirawan: So Richard, as part of my last question, I would love to ask you this question I call the three technical leadership wisdom. So if you can think of it just like an advice that you want to leave to us, the listeners here to learn from you, what would that be?

Richard Rodger: You have to let the developers that work for you make their own mistakes. And you have to create a culture of making mistakes blame free. So that means sometimes as a senior or the architect allowing code in the system that it’s maybe not the of the quality that you would write yourself. And I think a great deal of stress, especially for new team leaders and new architects, comes from this tension where it’s literally impossible to get everybody to write code to the company. Maybe you think you’re good, I don’t know, right? But it’s whatever your perception is, you’re never going to get other developers to write code the way you want. And I think that’s why components are so important, because if something is written one way inside a component, there’s less, I think the blast radius is smaller.

People talk a lot about empathy and they use the word empathy a lot. But I think in technical leadership, a lot of that empathy is showing leadership by accepting a diversity of technical perspectives. And allowing people to maybe make a mess. I’d much rather, if the developer came to me and you hear stories of people who’ve like destroyed, deleted the production database. Ah, you’ve told, like I would hire that person without thinking. I wouldn’t even look, I wouldn’t even think about it. You know, I don’t even want to see your code, right? You delete your production database. I’m going to give you a job.

That experience is something that you can’t rep. I mean, I’ve deleted live websites. You can’t replicate that level of experience, but for an organization to appreciate it, you’ve got to have empathy. Right, instead of looking for that kind of blame culture, so yeah. Go easy on the PRs, is how I would sum it up. Doesn’t really matter.

Henry Suryawirawan: Anything else or that’s all? I think we can also wrap up with that.

Richard Rodger: I think that’s the most important thing.

Henry Suryawirawan: All right. So thank you so much for the wisdom. I think it’s really powerful and beautiful, right? I like the empathy side. Obviously, as many architects or tech leads, sometimes we are very strongly opinionated about certain stuff, be it architecture, design or even coding style, right? So sometimes it all doesn’t really matter as long as you know the right outcome that you want to achieve.

So Richard, if people want to follow you or they want to chat with you about all these different perspectives about microservices, is there a place where they can find you online?

Richard Rodger: I am RJRODGER, still on X/Twitter. I’ve started using Mastodon, things like that. Same username, but yeah, Twitter’s still where it’s at by the looks of it for the moment for tech. And uh, yeah, the microservices, Reddit, places like that.

Henry Suryawirawan: Right.

Richard Rodger: Henry, thank you so much. It’s been super fun.

Henry Suryawirawan: Yep. Thank you so much for this time as well. So I hope people enjoy our discussion about microservice today.

Richard Rodger: Alrighty. Thank you. Take care. Bye bye.

– End –