#53 - Principles for Adopting Microservices Successfully - Chris Richardson

[00:02:14] Episode Introduction

[00:02:14] Henry Suryawirawan: Hello again to all my listeners. It’s great to be back here again with another episode of the Tech Lead Journal podcast. Thank you for spending your time with me today listening to this episode. If you haven’t, please follow Tech Lead Journal on your favorite podcast app and our social media channels on LinkedIn, Twitter, and Instagram. You can also make some contribution and support this podcast by subscribing as a patron at techleadjournal.dev/patron, and help me to continue producing great content every week.

For today’s episode, I’m extremely happy to share my conversation with Chris Richardson. Chris is a renowned thought leader in microservices, and the author of the book “Microservices Patterns”, and the popular microservices.io website. He also regularly speaks at international conferences and delivers consulting and training that helps organizations to successfully adopt and use microservice architecture.

Microservice architecture has been around for a decade now, and it has been widely coveted as the modern architecture to solve any kind of distributed scalable system. And much debate has also been discussed on how to adopt and implement microservices properly, including how to migrate from a monolithic architecture to microservices.

In this episode, Chris and I opened our conversation talking about the current state of microservices versus monolith architecture. And Chris explained why he thinks monolith is not actually an anti-pattern, and when it’s a good time for us to consider adopting microservice architecture. He then shared about the success triangle for implementing microservices, important concepts, such as design time coupling, and some important microservices patterns, such as eventual consistency, the Saga pattern, and how his current work on Eventuate can help developers to implement these patterns easier. Towards the end, Chris briefly explained some of his important principles that we should consider for decomposing a monolith successfully.

I highly enjoyed my conversation with Chris, learning about microservices and how to adopt an implement it correctly, and I believe that you will enjoy this episode as well. Consider helping the show by leaving at the rating, a review on your podcast app, and you can also leave some comments on our social media channels. Though it may seem trivial, but those reviews and comments are one of the best ways to help me get this podcast to reach more listeners, and hopefully they can also benefit from all the contents in this podcast. Without further ado, so let’s get this episode started.

[00:04:56] Introduction

[00:04:56] Henry Suryawirawan: Hi, everyone. Welcome back to another new episode of the Tech Lead Journal. Today I have with me someone that I really admire. He’s one of the thought leaders of microservices. He’s actually the original founder of cloudfoundry.com. He’s also a recognized thought leader and speaker in many different conferences. He is the author of the website called microservices.io. So if you are into microservices and have researched microservices before, maybe you have bumped into one of his articles or blog or patterns from that website. He’s also the author of the book “Microservices Patterns”, and he’s currently working on his startup called Eventuate. So maybe we’ll also ask a little bit more about what Eventuate is. His name is Chris Richardson. I’m really looking forward for this conversation, in fact, to learn more about microservices. So welcome Chris to the show.

[00:05:48] Chris Richardson: Great. Thanks. I’m excited to be here. Thank you for inviting me.

[00:05:52] Career Journey

[00:05:52] Henry Suryawirawan: So Chris, maybe for people who are not familiar with you yet, could you maybe introduce yourself, telling us more about your career journey, any highlights or turning points?

[00:06:01] Chris Richardson: Yeah. Maybe I’ll just start with the end. So for the past, gosh, it’s starting to be forever now, I guess the past seven plus years, actually, even, maybe even eight years, I’ve been somewhat focused on the microservice architecture. I used to say I like to travel all around the world, helping organizations adopt microservices through mixture of consulting and training. Though with COVID, I just do Zoom from my home office. Fortunately, businesses transitioned online, but I really am looking forward to working with clients face to face. So that’s what I do these days.

How did I end up here? It was a long journey, right? When my first paid job actually was the summer before college back in 1982. It’s funny. When I was a teenager, I was actually interested in programming languages and compilers, and so that was my focus area. And after college, I got a job with a company building Lisp systems. So if anyone can remember Lisp. That language with all of the parentheses, which at the time was used very extensively for AI. People were building quite complex systems. I remember one client or one customer. They’re actually using a Lisp based system to schedule time on the Hubble space telescope. And then another client was actually using Lisp to do payload planning for the space shuttle. So it was actually used for some super interesting applications. Anyway, so I spent seven years implementing Lisp systems, building everything from like compilers, garbage collectors, all the way up the stack to rich sophisticated IDEs. And this was actually all on machines that had 8 MB of memory. I feel like you could run it on your smart phone today, right?

So anyway, that sort of gave me an appreciation for sophisticated modern languages, because certainly at the time, late eighties, early nineties, mainstream was C programming. Here I was building software in this sophisticated object-oriented, functional language with garbage collection. Then, of course, Java came along, right, that was 1996. And initially I looked at it and thought, ah, what a primitive language, but ended up embracing it. And so I’ve been in the Java community ever since, basically. I was in the consulting group, helping clients build applications on top of Web Logic platform. And then discovered the Spring framework. I went to the ServerSide conference. I think it was Las Vegas 2004. Discovered the Spring framework and Hibernate and those technologies, and just immediately switched to that. Literally, I came back to work the following Monday and just said, okay, we’re going to switch to Spring and Hibernate. I was this architect on this mobile device management platform. And we were just like, okay, we’re going to migrate away from EJBs and that technology to this more modern technology, which remarkably, 17 years later is still dominant in the Java space. And that actually led me to write my first book, which is “POJOs in Action”. That was all about building applications with Spring and Hibernate, sort of transformative enterprise Java technologies. That was 2006. And I did ended up doing consulting and training around that for a while.

And then I discovered the cloud. Actually, that was 2006. An evangelist from Amazon gave a talk at the local JUG. We thought he was going to give a talk about APIs for buying books. Instead, he talks about S3, SQS, and EC2, which just blows my mind. It’s like, wait with an API call, you can provision 20 servers and pay 10 cents an hour? So I just got super excited about that and ended up getting, I guess it was beta access, maybe a few months later, 2007. And that led me on the path to creating cloud, the original Cloud Foundry. That was a Java PaaS, few clicks of the mouse, you could upload your WAR file, and it provisioned a bunch of EC2 instances running Tomcat, Apache, MySQL, and monitored, managed, autoscaled. Remember none of that existed in AWS functionality at the time. And this actually was helped by the financial crash in 2008, which made my consulting business evaporate. And it was like, well, I got this time on my hands. Instead of a zero revenue consulting company, I can be a zero revenue early stage startup. Much better sounding, and that led to the creation of the original Cloud Foundry, which was then acquired by SpringSource. I reconnected with the Spring folks, which was pretty awesome, and then we got acquired by VMware, and then spun out into Pivotal.

And I should say for clarity, today’s Cloud Foundry was developed by an entirely separate group, but they took the name from the Cloud Foundry that I developed. So I have no claims on the technology. But it’s nice that at least the name lives on. During my time at VMware, I got interested in this concept of basically functional decomposition into a set of services. And that altered that architectural style, which actually is a very old one, ultimately got the label of microservices. I got interested in that sort of 2010, and then just got more and more interested in it, and started talking about it in 2012, and it generated a lot of interest. Gosh, here I am, 2021, still talking about the same old stuff.

[00:11:50] Henry Suryawirawan: Wow. Thanks for sharing your story, the history where you got to where you are at this point in time. So thanks for sharing that.

[00:11:56] State of Microservices vs Monolith

[00:11:56] Henry Suryawirawan: So obviously, you have been doing this for quite some time. You mentioned 2012, maybe it was your first talk about microservice. Now it’s almost 10 years. So what do you think is the current state of all this microservice versus monolith debate? Or what kind of, you know, the cool things about microservices this day, if it’s still cool?

[00:12:14] Chris Richardson: Yeah. Well, I think almost every technology goes through the Gartner Hype Cycle, where first it’s amazing, and it can do no wrong, everybody should be using it. And then people realize that no technology is perfect, right? Nothing is the silver bullet. And then there’s the backlash, and it’s like, oh, that technology sucks. And then ultimately people figure out the trade-offs within, and the areas when it’s best to use microservices or a technology in general. I think very much, microservices is going through that hype cycle right now. And there’s sort of numerous anti-patterns of adoption. People are using it when they shouldn’t. It’s like I’m going to upgrade to a modern technology stack, and automatically assume that includes using microservices. Usually, there’s a whole bunch of criteria for when you should use any particular technology versus another. And that’s exactly true with microservices as well. The monolithic architecture is great for many scenarios, and then the microservice architecture is great for other scenarios. Like with everything, it’s a giant “it depends”.

You could certainly say if you have a large project with a large number of teams. Given that one of the primary goals of microservices is the rapid, reliable, frequent delivery of software, and the idea is, so in order for an organization to compete successfully in the modern world where new competitors can emerge out of nowhere. I actually think it was Hong Kong. Government decides to license six digital banks. Suddenly, the brick and mortar banks have six new competitors. Presumably, they need to be able to react quickly to that. And then, of course, COVID, you can say that’s like the ultimate disruptor. Suddenly in the US, online grocery delivery jumped forward five years. Actually, within about three weeks, I think it was. The world is very dynamic and unpredictable, and businesses need to be nimble, which means that IT needs to be nimble, which then translates into certain architectural requirements. Testability, deployability, modularity is important, so that you can rapidly deliver changes to your software. And sometimes, some situations you can have a modular, a monolithic architecture with those characteristics. But once you get to a certain size and certain level of complexity, quite often, the microservice architecture is a better fit.

It all very much depends. But you could say that, you know, if you look at the world’s leading technology companies, I would think the majority of them are using microservices. Amazon basically adopted, I’m going to say, a distributed architecture back in 2002. Google have a distributed architecture. Well, those companies are operating at massive scale with a gazillion users. But certainly, in many cases, they didn’t adopt services because of scaling per se. Certainly, this is the case with Amazon. They adopted them because they were operating in a highly competitive environment and they needed to move quickly. Because you can scale a monolith more or less to have, you know, support a large number of users. The microservices is all about tackling complexity.

[00:15:43] Monolith is Not an Anti-Pattern

[00:15:43] Henry Suryawirawan: I saw one of your articles about monolith, which I think is pretty interesting to maybe educate or clarify with a lot of people. So in that article, you basically title it, “Monolithic Architecture is Not an Anti-Pattern”. So while so many people these days think, okay, monolith is a bad architecture, we should move away from it, should adopt microservices. Why would this article saying that it’s not an anti-pattern? Maybe you can enlighten us on this front.

[00:16:09] Chris Richardson: Well, I mean, if you think about what a monolith is, it’s a traditional architecture where everything gets packaged, as say in the Java world is a single WAR file. So you architect it from a logical perspective. You’ve got modules, packages, and they’re collaborating to handle requests, and you’ve got a single database. I mean, it could be a bit more complicated than that, but it’s best think of it as a single database. That’s really simple. All communication within between the modules of your application are simply method or function calls. All of your data is in a single database, which means that you can just use regular ACID transactions, and this is the application. So, it’s just simple and familiar. Whereas immediately switch to the microservice architecture, you’re building a distributed system. So communication between the different parts of your application, and now some kind of interprocess communication. Whether it’s REST or gRPC or messaging and so on, which is just much more complicated than simply calling a method. And then your database. In a microservice architecture, one of the key essential characteristics of a service is that it’s loosely coupled.

There’s a couple of different meanings there, but one of them is design-time coupling, which is where you want to be able to change your service without other services having to change in lockstep. One key way to achieve that is to treat each service’s databases private. So it kind of like private fields of a class, right? No one can access that. So what does that mean? Well, it means that each service has its own database, so that makes transaction management much more complex. You can’t simply begin a transaction, update arbitrary data in multiple services and commit that transaction. Well, you could try, you could use two-phase commit, but that’s quite complex. And it also introduces another type of coupling, runtime coupling, which is where services have to be up together in order to handle a request. And that actually impacts your availability. So you end up, you can’t use classic distributed transactions, and you have to use eventual consistency, which is much more complex. So it’s like the monolithic world, simple, the microservice world can be more complex.

[00:18:46] When to Adopt Microservices

[00:18:46] Henry Suryawirawan: So hearing what you say about microservices and all this complexity, probably, it’s an exponential difficulty, in terms of difficulty, right? Where you have to handle transactions, for example, eventual consistency. How about failures? If you need to coordinate multiple services. So why suddenly all these craze about microservices? What do you think will be the right time for any company or for any product to actually embrace this?

[00:19:11] Chris Richardson: Well, it’s sort of you assume that you’re operating in a context where you need to deliver software rapidly, frequently, and reliably. You view the state-of-the-art for software development practices. This might sound extreme, but the ideal world is where I’m a developer, I commit my change, that changes just automatically built, tested, and deployed into production. And as a developer, I’m committing changes at least once a day. So you’ve just got this constant stream of small changes, automatically built, tested, and automatically deployed into production.

One of the challenges, so if you’ve got a small monolith, you can do that. There’re many different factors to consider. But if you think about, well, how long does it take to run the tests? That’s one key factor. You know, if your monolith is small, the tests will execute very quickly. So, your deployment pipeline can keep up with the rate of commits that are coming from your developers. As your application grows in size, it gets bigger and bigger, and then the test takes longer and longer. If you think about even just something as simple as the startup time for your application, massive monolith can take many minutes to start up, and then you can run some tests. So ultimately, there are these issues around the size. Just the pure size of your application slows down your deployment pipeline.

And then also from the perspective of a developer. If I’m working with this complex code base, that’s can be overwhelming as well. And there are ways to modularise it so that you can break things up. You now have a modular monolith where you can break things up into vertical slices, and which can be worked on independently. But ultimately, they have to be assembled together and tested. At some point, it becomes problematic. Plus, when you deploy the application, you’re deploying it in its entirety, which I think is just inherently complex. And then you contrast that with the microservice architecture, where you have a bunch of teams-sized services. So I’m working on the order management team. I’m responsible for the order service. That’s a much smaller, more easily understood code base. I can build it, test it quite quickly, push it into production, and I can change the order service without having to update customer management or account management and so on. So you have that independent deployability, and so it can be, if you’ve got a rapid rate of change, the microservice architecture is much faster, much more fluid.

And then the last point I would make is, you know, you think about modern application or successful applications that run a business, they tend to last for decades. So you imagine that you’ve got some 15-year-old application, and it’s massive. Upgrading that application to a new technology stack, even basic stuff like, I want to apply security patches. So, which means that you have to remain reasonably current. You want to hire people. They don’t want to work on ancient technology. So it’s important to keep your code base current in terms of what technology stacks it’s using. That’s really difficult if you have a massive monolithic application. Whereas if you contrast that with the microservice architecture, each service is relatively small. It can have a different its own specific technology stack, and you can upgrade one service at a time, and that helps you build these applications that are sustainable over the long run.

[00:23:04] Microservices Success Triangle

[00:23:04] Henry Suryawirawan: And I think you also mentioned there’s this triangle for success implementation of microservices, right? Could you maybe share what is this model about? What is this success of a triangle?

[00:23:16] Chris Richardson: Oh yeah. So the argument goes this way, right? Because the world is volatile and uncertain, businesses and hence IT need to be nimble and agile, and they need to deliver software rapidly, frequently and reliably and sustainably over the long run. And so the question is, how do you do that? And so that’s the success triangle. It says, in order to accomplish those software delivery goals, you need to do three things. One is you need to embrace DevOps. As defined by the “DevOps Handbook”, it’s this set of principles and practices for delivering software rapidly. Part of that is stream of small changes flowing into production.

The second part of the success triangle is the organization. The idea there is that you structure your organization as a set, as a network of loosely coupled, autonomous cross-functional teams. Because you really want to kind of avoid the silo-ed developers develop, QA tests and production manages, where you have to hand-off software from one to the other, and then the people are incented for very different things. Developers are paid to change things and Ops they’re basically paid to keep things stable. In other words, not change things. And so teams just have all of those responsibilities within the same team. Then actually operations, they provide platform to enable the teams to do self-service deployment. Excellent book on that called “Team Topologies”, which is sort of like a blueprint for modern organization design.

So you’ve got process, you’ve got organization. And then, the third part is architecture. You need an architecture that supports DevOps, and that’s getting into deployability and testability. You need an architecture that supports the organizational structure. And that’s where Conway’s Law applies. Basically Conway’s Law says that the structure of the architecture and the structure of the organization that created the architecture basically mirrors of one another. So if you want a loosely coupled organization, you need a loosely coupled architecture modular. Your architecture is comprised of modules, modules have APIs. There’s minimal coupling between those modules. So that the teams are not constantly in meetings to discuss and align, and all of the other things people seem to spend their time doing in tightly coupled organizations. You’ve got an architectural requirement. I think I said, testability, deployability, loosely coupled modular architecture. And then if you throw in the sustainable piece, that means you need an architecture that lets you upgrade your technology stack easily over time. Which, in a sense, you could say, is a modular concept sort of related to modularity or it’s a evolvability, right? So you’ve got a set of requirements that your architecture must satisfy. Which then goes back to, sometimes, you can satisfy those requirements with a monolithic architecture. But once you get to a certain level of scale, you’re probably better off using the microservice architecture.

[00:26:40] Design Time Coupling

[00:26:40] Henry Suryawirawan: So you mentioned a couple of times already regarding independent deployability, right? Modularity and also loosely coupled services. And you refer to this as design time coupling also a few times. What do you think is a design time coupling? What is that, actually? And why we should avoid that?

[00:26:58] Chris Richardson: Well, the degree of design time coupling between two modules, let’s say, A and B is the likelihood that A and B have to change in lockstep. And you want to minimize that, right? So let’s imagine, for instance, those two modules, whether it’s just say order management and customer management, they’re owned by different teams. So a requirement comes in, some new requirement comes in. The ideal scenario is that requirement just impacts one of those modules. So it just gets assigned to the relevant team.

Now, if there’s design time coupling between those services or between those modules, team A changes their service that actually forces a change to service B, which requires having meetings and coordination with the other team. Suddenly, the whole process of implementing that change takes a lot longer. Ranging from you actually have to find the time when you can both meet. You have to find a meeting room. Most companies I visited, they never had enough meeting rooms. It was like finding a meeting room was a major challenge in the work environment, which is sort of a sign that the organizations were too tightly coupled. Also, different teams can have different priorities, and so to get something done that spans teams, they both have to make it a priority, which can be challenging as well.

Whereas if work is happening inside a single team, you just have a discussion at your daily standup. People have done studies. ThoughtWorks did a study of this, and it was like decision-making across teams took 10 times longer than decision-making within a team. So it all largely ties back to the concepts of design time coupling. And so it’s sort of like, how do you achieve that? And part of it is having well-defined stable APIs that encapsulate the design decisions of the implementation so that they can be changed without impacting the API. So it’s very kind of core concept of software development that dates back to like the seventies. That’s why design time coupling is really important to just minimize coordination and communication between the teams.

And I’ll give you an example. This is from back in 2002. Nothing to do with microservices, but I was working on this device management platform. You know, I was leading the server team, and then there was a client team. They wrote low level code that went in mobile devices. We were in the US and they were in the UK. While we met, we agreed on the API, simple API that we were going to use to communicate. They just went off and did their development and we went off and did our development. And then let’s just say, six months later, we met and integrated the two pieces together. As far as I remember, it all just kind of work. And I think that was a good example of having loosely coupled software that could be developed independently. Otherwise, the opposite would have been some tightly coupled software where we’d been constantly having transatlantic phone calls. It would’ve just been a lot slower and a lot more unpleasant to actually develop the product.

[00:30:27] Henry Suryawirawan: So also I see a lot of people who started from monolith, because they are so used to have these functions all available to across the different teams and they just call each other. One of the common anti-patterns I see as well is they break into different services, right? Not necessarily a lot of microservices, they will tend to have this shared library. Maybe call it a core common modules that is shared across different services. So what do you think about that? Is that also a design time coupling?

[00:30:55] Chris Richardson: Well, it depends. You could view it as there’s two types of shared libraries. So one, libraries where different services can use different versions of the library and still function correctly. And that’s good. It just saves duplicated code, duplicated development. And that’s primarily suited to utilities. Then there’s the other type of library, and that would be a library that has business logic. Everything is great until the requirements change, and that library has to be updated, and all of the services that use it have to be running on the same version. Now, once in a while, that’s okay. It might be a worthwhile trade-off. Because if a library is embedded within a service, it’s invoked by a local method call that’s super efficient. So there’s actually a benefit to embedding, to having a shared library. But then the downside is having to upgrade multiple services in lockstep.

So it’s generally a bad idea, except when it’s not. You know, there was a company, Segment. I think they gave a talk at QCon about how we adopted microservices. It went badly, and we went back to a monolith. And I suspect that, if I remember correctly, the actual problem there was, they had a shared library that implemented business logic that kept changing. That forced their services to be updated in lockstep constantly. That scenario is known as the distributed monolith anti-pattern, which is basically where you have a distributed system, but you have to change all of it in one go. So you’ve got a monolith, and it really is like the worst of both worlds.

[00:32:47] Henry Suryawirawan: I think the keyword here is the lockstep. If you notice that you always have to do this kind of lockstep where you require one service to change before you can change, I think that’s anti-pattern that we should avoid, right?

[00:32:59] Chris Richardson: Yeah. That’s very much an architectural smell. So you should design upfront to eliminate it. But then if down the line, you see that it’s constantly occurring, where services A and B are regularly changing for the same underlying reason, that’s an indication that a refactoring is needed.

[00:33:21] Distributed Transaction and Saga Pattern

[00:33:21] Henry Suryawirawan: So another thing that is complicated in all these microservices architecture is what we deem for distributed transaction kind of thing, where you actually have to maintain consistency by committing in multiple databases across services. And actually your company, Eventuate, is doing a solution around that. But before that, could you maybe share with us, how do we commonly tackle or solve this problem?

[00:33:44] Chris Richardson: Oh, yeah. So the example I use all the time is customers and orders. There’s a requirement where the customer has a credit limit, which means that in order to create an order, you have to reserve some of that credit. So you could imagine the customer entity has an available credit attribute that gets reduced by the order total when an order is created, but it can never go below zero. So monolithic application, no big deal, right? In a microservice architecture where you’ve got customers and orders and different services, the order service has to reserve credit inside the customer service. That request spans services. The approach that you would use is the Saga pattern. It would go basically like this. So, step number one, the order service would create an order in the pending state. A message would be sent to the customer service, asking it to reserve credit. When it tries to do that, two possible outcomes, of course, well actually three, customer doesn’t exist, credit was successfully reserved, or the credit limit was exceeded. So the customer service will send a message back to the order service. And the order service, it will change the status of the order to “Approved”, if the credit reservation was successful. Otherwise, change the status of the order to “Reject it”, if it was not. So it’s a bit more complicated.

And then there’s a whole bunch of subtleties, right? We’re no longer using ACID transaction. It’s eventually consistent. But if you think about the context, imagine that the order service had a lot of complexity. The customer service had a lot of complexity. By splitting them into two, which would enable each one to be built, tested, and deployed independently, we’ve got tremendous benefits. It’s just that the downside is we have to use the Saga pattern. Depending on the situation, that can be a very worthwhile trade-off. I mean, once again, this is all a giant “it depends”. You know, Saga have the potential for them to be extremely complex, and it could be a sign that the two services should be merged. But in other situations, that can be worthwhile. You think about a really simple Saga, right? You have an order taking system that takes an order, and then it just notifies the fulfillment system, “ship this order”. That’s actually an example of a really simple saga. And no one would argue with that. It just works. One of the key things is sometimes, it’s very much a worthwhile trade-off. Whereas in another situation is, “Ah, it’s too complicated, we can’t do this and we have to merge together.” Yeah.

[00:36:36] Eventuate

[00:36:36] Henry Suryawirawan: And then how does your product, Eventuate, actually help with all this?

[00:36:41] Chris Richardson: Oh yeah. So basically, it’s a framework, or it’s a platform because there’s this sort of code that runs inside the service itself. And then there’s a supporting service that implements both the saga pattern, and then some of the key underlying patterns that make all of this work reliably. So you can build sagas. That’s definitely one part of it. Some of the underlying patterns are actually super interesting. So like one problem you have, is I said, well, the order service creates an order and sends a message, via the message broker. Those two things have to be done atomically. You’re doing two things, updating a database, talking to a message broker. So how do you do that atomically? Cause if you do one and not the other, your system is in a permanently inconsistent state.

So the pattern for that is actually known as the transaction outbox pattern. And that’s where instead of sending a message directly to the message broker, the service actually inserts the message into an outbox table in the database as part of the transaction that creates the order. And that gives you atomicity. Those two things because of the database transaction will both happen or neither of them will happen. And then there’s a separate process that’s pulling the messages out of the database and sending them to the message broker. And that’s sort of one of the key foundation patterns that Eventuate supports, because it’s actually quite complex. In the ideal case, rather than querying the database table, it’s actually reading the database transaction log, sort of highly database specific. So Eventuate does that. And then on the receiving end, the message broker can deliver messages multiple times. So the message handler has to detect and discard duplicate messages and Eventuate implements that pattern as well. So the idempotent consumer pattern. So that’s just a sampling of some of the key patterns that Eventuate supports, but the actual original motivation for Eventuate was, it was an implementation of the transactional outbox pattern, and it grew out of that.

[00:39:00] Tips for Implementing Saga Pattern

[00:39:00] Henry Suryawirawan: So based on the few things that you have mentioned, transaction outbox pattern, idempotent, receiver pattern, is there any other thing or maybe tips, for people to implement this saga pattern correctly?

[00:39:12] Chris Richardson: Well, it’s actually quite a deep topic. So, for instance, if you think about one of the interesting things about the saga pattern, right? So it’s a series of local transactions in multiple services. So if step five of a saga fails, you actually have to undo the changes that were done by the first four steps. And usually it’s a semantic, can be a semantic undo. For instance, so you actually have to execute what is known as compensating transactions to undo what was done previously. In the case of the customer and order example, changing the status of the order to reject it is actually a compensating transaction. So there’s a lot of careful design required. Once again, you contrast that with a regular ACID transaction, where the service can simply execute a rollback statement to abort the transaction. Really simple happens automatically. But with sagas, you have to design it. That’s another level of complexity.

The last thing that you have to actually think about when designing sagas is what happens when two sagas execute concurrently. Now, if you go back to like database theory, it goes back to the definition of ACID –atomic, consistent, isolated and durable. Turns out the sagas lack the isolation property. Then you go, what’s isolation? So isolation is a property, that means that the concurrent execution of two database transactions is equivalent to some sequential ordering. So if A and B are executing concurrently, the outcome is either A followed by B, or B followed by A. Primitive databases, they do locking and stuff to make that happen. But anyway, sagas don’t have the isolation property, and that gives rise to what is known as a data anomaly.

In classic database literature, there’re dirty reads where one transaction can read the uncommitted changes of another one, lost updates, and fuzzy reads. So if you look at the customer and order example, the order is created in a pending state. That’s actually an intermediate state because the saga has not completed. The ultimate outcome is still being determined. And that’s actually a dirty read. And once again, this sounds really scary, and in theory could lead to sort of bugs and so on. But it’s sort of a level that you just have to be aware of this problem. So, for example, someone queries the state of orders, and this order that’s just being created and in the pending state, is that a real order yet? Well, it’s still being determined, but it’s like from a reporting point of view, maybe it isn’t. And then if someone tries in theory, someone could cancel it. But what does it mean to cancel an order that is still in the process of being created and we may or may not have reserved credit? So there’s some sort of complex issues there.

To solve them, you have to be aware of these design techniques, known as counter measures. Counter measure is a design technique that eliminates these data anomalies. And it turns out, the reason you create an order in a pending state is, that’s an example of a semantic lock. That order is locked at the application level, and it’s sort of a soft lock. But the application logic, any application logic, should actually look at the state of the order and go, “Oh, it’s pending. Therefore, I should do something different.” Perhaps if someone tries to cancel it, maybe the cancel should fail with a status saying, “Try again later.” So it’s something you have to be aware of, and design for explicitly, rather than just relying on the database to make this happen magically for you. Shameless plug. I have an online course that talks about all of these issues.

[00:43:25] Henry Suryawirawan: Right. So I’ll make sure to mention that in the show notes. So as we heard all these complexities, I think it’s really good to have these kinds of patterns. In fact, I think microservices.io try to have these patterns available for people to read about and understand what is the common problem that you see? And what are the typical patterns that you can use to solve this? So I think that website itself is gold for people who would like to implement microservice.

[00:43:49] Principles to Decompose Monolith

[00:43:49] Henry Suryawirawan: Another thing that I saw you advocate is actually when people try to decompose their monolith, there are 10 principles. So I know that we are running out of time. We won’t be covering all of them. But maybe if you can do in a kind of like rapid fire manner. Maybe five different principles, maybe let’s start with “make the most of your monolith”.

[00:44:08] Chris Richardson: Yeah, like principle number one is don’t. In the sense, try and make the most of your monolith. So, for example, yeah, I’ve thought I had this conversation a lot with people where it’s like, “yeah you know, our development process is really slow and there’re bugs. I think we should just adopt the microservice architecture and that will fix our problems.” And it’s sort of like, yeah, maybe microservices will solve your problem. But I would first look at actually optimizing your development process. So, for instance, a common problem with a lot of enterprises is they don’t do automated testing and relying heavily on manual testing. While if you put in place automated testing, that’s likely to have dramatically improve your ability to deliver software quickly, right? Fewer bugs, higher quality software, much more rapid delivery. That’s just one example of how you can improve your development process with your existing monolith.

Especially, when you’re a small team of people, if someone came to me and said that we’ve got a hundred developers and we’re having problems, perhaps it would be more likely that part of the solution would be to use the microservice architecture. But if you have a team of five developers, probably not. And it’s not black and white. It all very much depends. But the smaller the team, the more likely that the monolith is okay. And it’s your development process that’s lacking.

[00:45:46] Henry Suryawirawan: The second principle that I find interesting is that “success is improved velocity and reliability”. Maybe can you explain a little bit more?

[00:45:54] Chris Richardson: Yeah. This is a really good point. So throughout this, I’ve been talking about rapid, frequent, reliable software delivery. What is that? So there’s actually four metrics which come from DevOps that research has shown to be a good indicator of high performance. One metric is lead time. What’s the time from a developer committing, pushing their changes, checking in that changes to that change being deployed into production. High-performing organizations, the lead time is short. State-of-the-art, you could say, is 15 minutes. Basically, you’ve got a fully automated deployment pipeline that builds, tests, deploys each commit. Another key metric is deployment frequency. How frequently are you deploying? And it’s somewhat tied in with lead time. You could say a goal would be at least once a day per developer, which is quite different from the, we deploy every two weeks to every once a quarter. So those are the two metrics that are measuring velocity. Well, I should say with deployment frequency, one insane example is Amazon were doing it like 130,000 changes a day to production, which is like 11.6 seconds.

And then there’s reliability and interestingly, you think about the traditional approach to doing things reliably is to do things slowly and carefully. But it turns out, and this is key book to read is the “Accelerate” book by Jez Humble and his colleagues. And they found that high performing organizations are delivering small changes very rapidly in production, and it also leads to greater reliability, which is completely counter-intuitive, right? But the idea here, if you do things slowly, that means every deployment, you’re changing a massive amount, which is actually highly risky. You don’t fully understand the impact of the change. Whereas if I change one line of code, there’s a pretty good chance I know what that’s going to be impact of that, the other extreme. So they are the metrics around change failure rate. How often do deployments actually break production? And interestingly with Amazon, with their massive rate of change, they’re hardly ever down. I mean, it’s really unusual to go to amazon.com and for it to not work.

And then related to that is mean time to recover. If there is a problem, how quickly can you detect it, diagnose it, and fix it. And once again, you should rarely have production outages, and you should be able to recover from them really quickly. Those four metrics, that’s the measure of how performant your organization is. So the whole point of microservices and adopting microservices is not to have microservices. It’s really is to improve those metrics. You know, I’ve worked with an organization where this is one case CIO says “do microservices”, and it was a top-down hierarchical organization. I’d meet with developers and I go ask, “Why are you doing microservices?” And they go, “Well, my manager told me to.” And it was almost like microservices became part of their KPIs. But that’s not the goal. The goal is not to have microservices. The goal is to improve those metrics, and that’s what people should be incented to improve. Because sometimes changing your development process around your monolith will improve those metrics. But then, in other situations where you really have outgrown your architecture, your monolithic architecture, then adopting microservices is the only way to improve those metrics.

[00:49:50] Henry Suryawirawan: So thanks for reminding us again the key metrics. It’s not how many microservices you have. How small is your microservice? So it’s more about all these four key metrics, DORA metrics, some of them say. So thanks for reminding that. Unfortunately, we cannot go on to cover the remaining eight, but for people who are interested in these principles, go check it out on Chris Richardson’s blog. I think it’s really insightful.

[00:50:12] 3 Tech Lead Wisdom

[00:50:12] Henry Suryawirawan: So, Chris, I know we are running out of time. But before I let you go, I normally ask this one question for all my guests, which is to share your three technical leadership wisdom as a tips for all of us here in our career. Maybe we can learn from you.

[00:50:24] Chris Richardson: Yeah. Well, I had a little time to think about it. I got a few things come to mind. I think one is just learn, be curious. The two other ones that sort of come to mind, one is it depends. Recognize that it depends. There are no silver bullets. You want to look at every technology as having a set of trade-offs. And this sort of goes back to patterns, benefits, drawbacks, and issues, which is sub-problems that you have to solve. And so when you’re doing any kind of design, you really want to just be aware of that. Lastly, I would just say, think. Evaluate the trade-off.

[00:51:07] Henry Suryawirawan: So it’s really a timely reminder for those of you who are thinking of adopting microservices for not a good reason. Again, it’s been a pleasure. Chris, for people who would like to learn more about this, maybe follow you or contact you, where they can find you?

[00:51:19] Chris Richardson: Oh, well, a couple of places. There’s like microservices.io. So that’s where the patterns reside, presentations, code, etc. There’s also ChrisRichardson.net. So that’s my consulting site and training site. And then obviously eventuate.io. That’s my company.

[00:51:41] Henry Suryawirawan: So Chris, thank you so much for spending your time. It’s been a pleasure. I learned a lot from you. So good luck with all your talks, courses, and also the product itself, Eventuate.

[00:51:50] Chris Richardson: All right. Great. Thanks. I’ve enjoyed it.

– End –