#125 - Patterns for API Design - Daniel Luebke

[00:00:56] Episode Introduction

Henry Suryawirawan: Hello to all of you, my friends and my listeners. Welcome again to the Tech Lead Journal podcast, the show where you can learn about technical leadership and excellence from my conversations with great thought leaders in the tech industry. If this is your first time listening to Tech Lead Journal, subscribe and follow the show on your podcast app and on LinkedIn, Twitter, and Instagram. And for those of you longtime listeners, if you find my podcast helpful and want to show your appreciation and support my work, you can subscribe as a patron at techleadjournal.dev/patron, or buy me a coffee at techleadjournal.dev/tip.

My guest for today’s episode is Daniel Luebke. Daniel is a software architect and the co-author of “Patterns for API Design”. In this episode, Daniel and I discussed some API design patterns and best practices taken from his book. Daniel first shared the importance of understanding domain requirements for building APIs and several API and message best practices, such as API first design, how to design loosely coupled message exchanges, the trade-off between generic and specialized API operations, and the risk of exposing too much internal data model and logic in our APIs. Daniel also introduced the microservices domain specific languages (MDSL) as an alternative to Open APIs for specifying APIs independent of the technology implementation. Towards the end, Daniel explained the importance of defining the API lifecycle, how to support backward compatibility, and the different API versioning strategies we can use to evolve our APIs.

I enjoyed my conversation with Daniel, learning about API design pattern and best practices, specifically on some strategies we can use to manage API lifecycle and evolution. If you also enjoy listening to this episode, please share it with others, either your friends or colleagues, who can also benefit from listening to this. Also leave this podcast a 5-star rating and review on Apple Podcasts and Spotify. It will help me a lot to make this podcast discovered by other people on the platform. Before we continue to the conversation with Daniel, let’s hear a few words from our sponsors.

[00:03:38] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I have with me a co-author of one of the recently published book titled “Patterns for API Design: Simplifying Integration with Loosely Coupled Message Exchanges”. That’s pretty long. So Daniel Luebke is here. He’s one of the co-authors. And I’m really excited to talk about this, because building API is something really important, especially these days. And especially we will talk a lot about the message part of the API. So Daniel, welcome to the show.

Daniel Luebke: Yeah. Thanks Henry for having me on your show. I’m looking forward to discussing the book, the patterns, the messages.

[00:04:17] Career Journey

Henry Suryawirawan: So Daniel, in the beginning, I always love to start by asking you about your career journey. If you have any highlights or turning points that you think will be interesting for us to learn from you.

Daniel Luebke: Yeah, sure. So, I mean, I’m now 43 years old, so there have been some positions I’ve taken. It has been actually a long journey. So I started programming when I was in fifth grade, so rather early. So I got my first computer when I was 12, I think, back in the days. Well, compared to what we have now, a very lousy hardware. But nevertheless, I started programming very early and even in school we had a team of five classmates and we were developing a game which was really successful. So you could buy it in the store on a CD-ROM back in the days, and I think that was the first interesting point in my career, because it was one of the great successes. You were still at school and everyone said you couldn’t do it. But it was then about team dynamics. It was the only released game, because the team fell apart after that a little bit, because there were different interests and different recognition of how good everyone is. So that, I think, was the first turning point.

Then I started in German, it’s called Wirtschaftsinformatik, and it would translate to business informatics, which is a subject which is very German specific. So it comes close to information systems, I think. But it’s more computer science plus economics. During that time, I also worked in very database heavy programming, and did trainings, and yeah, did lots of computer science lectures. It was a very great time, actually. Although one thing I learned, there was that even though the company I did the projects and trainings had a profit margin of approximately 50%, if you have a bad CEO, you can still go bankrupt. Need to take care of your cash flow, even in private life. There was quite a lesson to see.

I come from Germany. I’ve studied in Germany. I did my Master’s thesis then in Barcelona in Spain, which was very interesting. And then I came back and did my PhD back in Hannover, my hometown, in computer science, and the topic was Service-Oriented Architecture. So really about what we nowadays call APIs and orchestration and choreographies. Back in the days, obviously, it was a very different tech stack. Again, I learned quite a lot there. And then I moved to Switzerland, because during my studies I didn’t manage to go abroad. Okay. It was one semester in Barcelona, or some month, at least. But I really wanted to stay in a different country. And originally I wanted to go to England, but it became Switzerland. Was planned for two years and was eight years there. So, obviously, it has been a great time. Still a great time since I met my wife.

So there, I was in large projects, very heavy on financial industry, obviously, in Switzerland, and much complexity of in-house IT landscapes. So the reality shock which people get when they start in large organizations. And as a consultant, it’s nice cause you move between different organizations, you learn many different ways of designing systems, different architectures, different preferences by different teams. So it was great. And then I moved back to Germany. Now I’m the CEO of a small development consulting company myself, which is also interesting, because now you need to do more coordination. You have people doing work you would like to do. But still, I’m interested in the technology. I’m still working in projects which is important for me. So I think you can only be good in technology and in technology consulting if you still have hands-on experience. Yeah. So, that has been my life in three minutes or so.

Henry Suryawirawan: It’s very interesting that you started to share your story about creating computer games. I think it is almost like every childhood dream to come up with a game, and you managed to publish and sell it in CD-ROMS. I still remember during back then, I used diskette, floppy disk, but CD-ROM is like the next generation. But I think it’s still cool to be able to sell games when you are a kid. So thanks for sharing that.

[00:08:26] Writing API Design Book

Henry Suryawirawan: So let’s start to your book. So you have had a lot of experience in the industry. You have written a thesis, worked in a large financial company. Why specifically writing an API design book? Because API has been around for many years, especially when you said about SOA, right? SOA. And there are so many books already published as well. So is there anything different that you try to convey or try to teach us from your book?

Daniel Luebke: Yeah, I definitely think so. So the story behind the book is a little bit longer. I need to go back some years. Olaf Zimmermann who’s the first author and the team lead of this pattern approach. I knew him for quite some years. We met in Hanover. He’s interestingly born in city near to Hanover. We hadn’t seen us for a while, and then we met back in Zurich. He was in Switzerland. I moved to Switzerland. He became a professor there, and he was thinking about, okay, so what’s his line of research? And he was at IBM before. So also a mix of consulting, but also doing his PhD. Well, I said, okay, I want to have this idea. We are building all these services we have. And then back in the day, microservices were really the hype. We are building all these and that, I need to teach to students, and I have corporations who would need to build API. So I want something to teach.

I, at the time, was new at a very large scale project, where they’re connecting land registries, banks, notaries, and some other parties across the whole of Switzerland. So it’s a system where they have like thousand different organizations connected and automating all these business processes. As you can imagine, there are many services, many APIs involved for doing that. A bank places a request, the platform generates some documents, sends them back to the bank so that they can digitally sign this. It goes forwarded to the notary and then to the land register, and whatever you can do there. There were some people who were very new to service design, and we had many discussions about things which I would’ve thought were basic but, obviously, they weren’t. So my motivation was more for my daily work life to just say, here’s the pointer. Please read this there and then we can have this discussion again.

Yeah, so that was, essentially, how it started. And then other people joined, like Mirko and Cesare and Uwe. I can’t recall now exactly in which order, but quickly became this team of five people. Uwe and Cesare are both very experienced as pattern authors already. Olaf and I and Mirko as well had some practical or more practical experience, although Olaf was a professor and Mirko actually is now as well. I’m not. I’m still in the industry. I think that’s the motivation, and then the question is, how do you teach these things? So I mean, design architecture is always about decisions, and decisions have these nasty aspects of that. You get something, but you lose something. If you go through the right door, you can’t go through the left door. That sounds easy, but people always assume that there is this perfect solution. You just do something and then it’s perfect. And you do this every time because it works all the time. The silver bullet thing, and we all know it doesn’t work. I think even if you are fresh from university, at least after three years, it doesn’t work that way.

And patterns make that explicit. I think that’s so great, because obviously you have a name and I think we’ll talk about names as well. But you have this section about forces, so what are the influences on your design, and some things are better if you use a certain pattern, some things are worse? So you pay for your decision. And patterns make this explicit and that’s why it’s so great as both a teaching tool and also reference tool. So even nowadays, if I’m in projects, it’s just nice to go through the pattern list and check whether there’s something I should have considered or not. Because you are always busy when you’re doing things. It’s nice to have checklists.

Henry Suryawirawan: I always love reading pattern books, because it is practical. That’s the first thing, right? And also it kind of like lists down the decisions or the tradeoffs, some people say, on why you choose a certain pattern. So, for example, if you opt for scalability, then maybe this pattern might be better, for example, microservices, right? But obviously we know that microservices are not silver bullet. In some cases, it might not work as well. So Fred Brooks has this write up last time, “No Silver Bullet”. So I guess it has been mentioned many times in any kind of architecture kind of discussion I had in this Tech Lead Journal episode.

[00:12:53] Loosely Coupled Message Exchanges

Henry Suryawirawan: Your subtitle of the book also mentions specifically about loosely coupled message exchanges. So tell us more why you choose specifically loosely coupled message exchanges as your subtitle?

Daniel Luebke: Well actually, the subtitle was one of the last things we decided. And if we look at the book, what’s special about that is it closes a gap between high-level books like Enterprise Integration Patterns, which are, well, not high level or very abstract, but they’re certainly on a high level of abstraction. And then you have these books about technology. So how do I use REST or gRPC or GraphQL or whatever? And what verbs do I use under which circumstances? How do I build good UIs and all these things? In between, there is something. So you have a conceptual API design. Something where you decide, okay, I need these kinds of operations. I need to have these quality attributes, and because of this I perhaps do pagination. I chunk things and not transfer a whole message. And that’s independent of the integration technology of the protocols you use.

Back in my first project, it was a banking system or a banking platform and they used CORBA all the way. It was all CORBA. Already there, there were many patterns which we still use today, and more REST HTTP APIs, because these are really long living, and even now, whatever comes up next you will have these problems and you will have these patterns, hopefully, addressing those problems. So it’s not tied to any specific technology, first of all. So you can make your API design on a conceptual level using these patterns and then choose your technology. So that’s one aspect. The patterns themselves are not coupled to any technology themselves. But obviously, it’s probably more about the coupling between API clients and API providers. I think a good API and it’s why nowadays, finally, we all agree that API first design is the way to go, is you don’t want to expose your internal data models or your internal logic too much on your APIs. And the more your clients are not under your control, the less you want to do that.

I think most patterns here will allow you to deal with this problem. So how do I build APIs, which, well, if I do them by the pattern book, so to say, although it’s not, your goal shouldn’t be to use as many patterns. But if you just follow these forces and things, then you will hopefully have a design which allows you to have these loosely coupled exchanges, which are not dependent on your internal models.

[00:15:31] API Design Best Practices

Henry Suryawirawan: So you mentioned something, API first design. I think this is mentioned many times. In fact, I think I have one episode, James Higginbotham, also talking about that. And in the book, you also mentioned a couple more good practices for API design. So I think these days almost all backend developers will create API, especially REST API, or maybe gRPC or GraphQL. Are there any other things that you would advocate for all these engineers? What kind of good practices they should do or they should consider when building API?

Daniel Luebke: Well, first of all, you should think about how to express that. If you start with any software development, it’s just not only APIs, but it’s in general, you need to understand your domain. I think that’s important.

The problem with APIs is, let’s go back to the specific project. You have a team of people who, if you want to build a good solution or a very good solution, you need to understand what banks are doing, what notaries are doing, what land registries are doing. The interesting part is in the last couple of hundred years, I mean, this domain is very old, houses and government management of houses for a long time. So land registries are around like forever, and banks are very old concept and notaries I think as well. And over these couple of hundred years, these three parties still haven’t learned what they are doing. So that was very interesting. A bank doesn’t know the data model of a land register, which is very interesting and surprising. So if you want to build APIs for a domain or for clients where you don’t know exactly their requirements, it’s the first thing where you need to reach out to clarify this, and that’s easier said than done.

There are a couple of approaches. So first of all, you’re doing workshops, building your first API version, release it, try to evolve it. But then you need to deal with lifecycle issues. So one part of this loosely coupled, is also you need to decouple the lifecycle of your client and your provider, because you won’t deploy a new version simultaneously. So you will have points in time, many points in time where there are different versions. So a client normally supports an older version, and the provider rolls out a new API version, which should be backward compatible, so the client doesn’t break. So that’s kind of the ideal scenario. Well, many people say, okay, just build backwards compatible APIs. Yeah, but it’s not always possible. Sometimes you have breaking changes. If you say, “Okay, I don’t know my clients well enough. I want to iterate and I want to improve things”, it’s more likely that you will have breaking changes.

For instance, banks usually had one-to-many relationships, where the land registry internally manages many-to-many relationships. These kinds of things if you know them later on, so you have your API defined with one-to-many, and now you need to support many-to-many. It’s likely going to be a breaking change. Also, in your environment, there might be breaking changes. So in Europe, we have the Euro as a currency, which before were multiple national currencies, and if you had online banking back then to money transfer, there’s no way of doing that now. Yeah. So it was a breaking change that there’s a Euro and not all these local currencies. The same goes for account numbers. So they have been local, national account numbers, and now we have the IBANs (International Banking Account Number). So there will be changes in your environment which will break where you can’t do anything, and there will be changes where you learn your domain better and where you can say, “Okay, in theory, I should have known this already”, but you didn’t. The more you iterate to get your requirements, the more you need to think about your lifecycle.

Some patterns we have there is, for example, we have an experimental preview pattern. So you say, okay, let’s say we roll this out and, currently, I don’t give you any guarantees. You can check this out, you can give us feedback, but it will break without us telling you. So that’s not nice for the client. But the client also has some benefits, because he gets a premium. So a better version where they can use the API, preview it, use it, give feedback, and perhaps, hopefully, getting a better API down the road due to this feedback, even if it has breaking changes. And, obviously, some organizations or some developers will say, “Okay, no. Under these circumstances, having only an experimental preview, I don’t integrate, I don’t use this API,” which is fair because it’s known. What’s unfair is if you say, okay, use an API, and then you just say, “I don’t support it anymore. I break it in some way or another”. So being clear about your evolution process, I think, that’s important. So have your requirements, know how good you know them, and then plan ahead. Cause in the beginning you can. So if you just had a guarantee like, we support this for three years and then you just try to alter this later on, people will just come to your office and hit you, probably. These are decisions with which you need to do as early as possible, and then, obviously, you need to flesh out all the domain knowledge in your API payloads.

Henry Suryawirawan: So yeah, it’s always wise to start with the domain understanding first. And in fact, things like DDD, Domain-Driven Design is also something that advocate you to actually understand the problem first rather than coming up with just the technology or the solutions.

[00:20:48] Message Best Practices

Henry Suryawirawan: People these days, whenever they build API, the first question that they have in mind is definitely which technology like HTTP, REST, gRPC, GraphQL, and things like that. But, actually, specifically in your book, you also cover the other important parts, which is the payload itself, you mentioned, the messages. Maybe for us engineers here, can you give an overview? What aspects that we should consider from the messages point of view, the payload point of view in order for us to design our API better?

Daniel Luebke: Well, yeah, I think the most important part is how many requests do you have and how large your message payload is. So if we look at that, obviously, you could make an API operation, which might be an HTTP resource. When you say, get me all, give me all customers, and then you get back our customers, and everyone’s happy. You can deal with the data whatever you like. This approach, obviously, has some drawbacks. So much load on the API provider, because it needs to fetch all these customers. You have much load on the network, because you need to transmit all these customers. So we don’t do this design. What we usually do is get a certain customer or search for customers.

But is that everything we need to consider? No, usually not. So if we look at a search, even a search might return too many records. That depends a little bit on your domain on how many objects you have or domain objects you have. So if we have a service which would return a list of currencies, we know how many there are, and it’s quite manageable. If we are fetching customers from any large organization, Apple, Microsoft, or the whole Salesforce database, whatever you like, that’s too much. So we need to have some mechanisms if we are running into this problem to limit our response sizes. So you can either say, “Okay, I only return the first hundred hits I have”, might be a good strategy. We could have pagination. So I give you the first hundred and if it isn’t sufficient, you can query the next hundred. We could also say, okay, perhaps you need not all customers, but we query all sessions in a conference, and that’s a very huge conference when you have like 600 sessions. To each session you have some information like the title. Okay. That’s just a string. Who’s giving the presentation? That’s just a string. The image of the presenter. So that’s obviously some large payload part. Would I put this in my message? It depends on the use case, but most likely not. I would just put a link to this image, and whenever the client probably pops into the detail view, then it can fetch, this data is required.

So we have this decision on whether we need to include data or whether we reference data. Depending on your use case, you often have things which you don’t need. Coming back to the Swiss land registries, you have hundreds of years of historical data and, for most use cases, you don’t need historical data. Yeah. So it makes sense in one way or another, and there are multiple options. Address what data the client needs, and you could do many operations. You could do some parameters, which there’s a pattern called wish list. So I say I want the historic data or not. So it’s just a boolean flag, and if you set it, you just get back 20 times the data like you would have if you have no historical data. And if it gets more complex, then you might have not only a wish list, but a wish template where you can define in your whole data structure and all the nested things, what data you want to have or not, which then closely or easily gets you to GraphQL. The whole purpose is just to specify a request with exactly the data you need.

And so, there are many things you can do to adjust or trade off the number of requests and the request size and the response size, especially. And I think that’s a point where we need to make more conscious decisions. On average, I think most people just dump out everything which they have, and then at a certain point they just see, okay, well, it doesn’t scale. The database load gets too heavy and things are getting too slow.

[00:24:51] Generic vs Specialized API

Henry Suryawirawan: Definitely, creating an API and designing it well. The first point that you mentioned is about data size, the payload size. Do you return everything or do you include the large payload as part of the first response? I think that’s always tricky. The other trade-offs that I normally see in day-to-day experience is that some people don’t know whether they should create specialized API. You know, things that work for certain use cases only, or for certain customers only, or they should create something more generic API. So is there any kind of consideration that you would do to decide such a thing?

Daniel Luebke: Well, that leads us to the interface segregation principle. And it’s hard. I mean, if you look at GraphQL, it’s one extreme answer, obviously. So just tell you, I have a very generic API, and you as a client can specify what I return. And that’s nice in a sense, because the client just fetches the data it needs, can specify it.

But on the other hand, it’s kind of a leaky thing, cause, you as a provider, you can’t optimize for certain operations. So if you don’t have an index on certain database columns, GraphQL request might be painfully slow and you cannot anticipate it. On the other side, if you have many specialized operations, you can very well optimize your database access patterns for those, or caching strategies, and all things. So you can better optimize, but obviously you have much more complexity, different kind of complexity. One is you need to implement and manage many different operations, and on the other hand, you need to implement and deal with the complexity of GraphQL and the implementation behind this.

So, again, there seems to be some middle ground. What I would recommend is to look at what use cases you have in general, which you anticipate, and then to check whether there’s a certain amount of, like two, three operations, which would more or less cover these well. If you have to have like 20 different ones, then you probably should move on to GraphQL or building something similar. And if you have just only one, then it’s fine. As a rule of thumb, I would probably just, after having three different specialized operations, I would rather check whether it’s an approach where it leads me down the rabbit hole of having 20 specialized operations in the future. But it really depends on the system.

[00:27:13] Exposing Internal Data Model

Henry Suryawirawan: All right. Thanks for sharing that because yeah, definitely, it is based on the context that we have, but it’s one of the struggle that we have. Another one that is a struggle is actually to expose how much from our internal. You mentioned earlier in our conversation, that we should be conscious not to expose too much of the internal representation of our data model. For example, database schema. So how can we come up with a better exposure of our domain model as part of the API request and response? So maybe a little bit of advice here as well, because I can see so many pitfalls people have gone into simply because they decide exposing data model unnecessarily. So maybe a little bit of advice here, Daniel.

Daniel Luebke: Well, I think, really, it’s what you would call API first nowadays. Or put it differently, you should have a separate API interface. Whether it’s open API or something else, well, in the end doesn’t matter. But you should do that. You should work on this interface without working in parallel on your code. I think that’s one of the pragmatic things, because if you’re just working on your code and you have just built your database tables and you are building all your persistence layer, all these things, then your mind is thinking in all these data structures. Well, you need to be a very good developer to then switch over and say, “Okay, on my API, I want to have a more conceptual view”. So it’s much easier if your brain is working in the conceptual mode, like you’re doing a DDD design. You are more involved currently in requirements discussions. In this state of mind, it’s much easier to build an API, which doesn’t leak your technical information. And I think that’s a very simple trick, but it works astonishingly well in my experience.

Henry Suryawirawan: Yeah. Actually it’s a very good advice because I can still remember those days when I, you know, used to code a lot. So every time as a developer, if you start from the code, definitely, it’s very hard to still think about good API design practices, right? Because normally, of course, we always start with the data. We create the schema and easily, we can just expose everything up to the API layers. So I think this is definitely not a good practice. So what you are saying, API first, definitely makes sense.

Daniel Luebke: Well, it should, and that’s also a thing, a state of mind thing. You should. And it goes hand in hand. You should think from a client’s perspective. And that’s also something which developers often do is you’re building a backend code, you’re just putting an API on top, and you’re doing what’s easy for you. Because you have the service class there or this repository is already there and you don’t think about whether this operation is placed in a logical, sensible place or whether the data structure is really what you need to have there or whether the interaction from the client side makes sense or whether the client needs to know too much.

I think if you look from more formal requirements or DDD perspective, closely resemble looking from the client perspective. And that’s also a state of mind. Yeah. So, again, in your provider implementation, you probably think about what you have there, what’s easy for you, but what you should really think about is what’s easy for the client. Because in the normal case, you have like hundreds of your successes, thousands of clients and only one provider. So you should optimise your development work of the clients and not of yourself.

Henry Suryawirawan: I think that’s a really good advice. So always think from the API client’s point of view, not as a provider, right? So thanks for reminding that.

[00:30:37] Microservices Domain-Specific Language

Henry Suryawirawan: I also noticed in your book that you came up with this thing called microservice domain specific language. Maybe you can elaborate a little bit more. What is this DSL specifically and why do you create it? What kind of problems that it tries to solve?

Daniel Luebke: Well, essentially, you should do a podcast with Olaf about that, because it’s his baby. Probably I might be wrong, so Olaf might correct me in some blog post or so. But I think there were two motivations for him.

The first one is to have or to make the patterns explicit in your API contract. So a pattern language, if we go back to this gang of four book, that’s probably one of the most successful books in programming. Not because people nowadays buy these books, because they don’t anymore, I think. But because the language is now deeply embedded into all of our programming languages and frameworks and things. And so, that’s why they’re successful as a language. And these names convey, well, the whole design concept just in a single name. And so if we have observer or listener, we know, okay, that’s that. And probably if we now talk about pagination, people know, okay, they have an idea on what this means. And so one motivation was to annotate your APIs in the contract with, okay, here, I use pagination, and that’s a public API, and we require an API key just as keywords of this interface language.

The other motivation was to specify an API more independently of the technology. So from MDSL generators, which would generate Open API, and they’re also bindings for SOAP services and things. So it’s independent of the technology. I think probably Olaf might, again, correct me, it’s much more concise than, let’s say, Open API. Because if you look at Open APIs, really, there’re many, many lines you need to express things, because every attribute goes into a new line. So you have your data type, it has a name, new line, it has some occurrences. New line. It has a type. New line. So these files are getting very long. Interestingly, XML schema was much more concise in terms of line numbers. I mean, both formats probably shouldn’t change in a text editor but use something which can deal with these formats. But line wise, XML schema was much more compact. MDSL has this compactness. It’s not XML, it’s not JSON, it’s more or less human readable language, which is very compact, where you can also insert all these patterns and where you can then generate code and ensure API interfaces. And for the book, it was nice because if we printed Open API, we would just have probably 500 pages more.

Henry Suryawirawan: Thanks for giving a summary of that. I’m sure if we get confused with the MDSL, I will have a special episode to discuss about this.

[00:33:23] API Evolution

Henry Suryawirawan: So, Daniel, I want to go through to the last section of API design patterns. Do you have any favorites that you want to cover?

Daniel Luebke: Personally, I was very involved in the evolution section. Perhaps we can have some at least general remarks there. Because I think it’s a section which is a little bit special, because all other patterns are very involved in the message contents itself. And evolution is a bit more out of the message. And I also have seen this many times at some companies. Back in the day, they said, okay, we’ve done our service-oriented architecture. The only thing missing is our versioning scheme.

First of all, versioning isn’t the important part in that sense. That’s just byproduct. What you haven’t considered is your life cycle strategy, and that’s probably one of the most important parts you have thought of first. People are always good at bringing new APIs, new services into the landscape, but they are very bad on getting them out of the landscape. So decommissioning APIs or changing APIs and managing the clients and the impact to the clients. So that was quite interesting that a large insurance company said, yeah, well, we are more or less done with that. But we haven’t thought about our life cycle. Yeah. Well, go back to square one. And that’s why I think evolution is important and there are not that many things you can consider. Essentially, have a question of how backward compatible do you want to stay? Which involves certain costs from you as a provider, which might make the API dirty over time. So we refactor our source code all the time to make the structure better, to make names better. Staying backwards compatible means that I can’t refactor many aspects of my API cause it would break.

So many people just put price in there, for instance, or some attribute names without the unit. That works if you don’t have a single unit, like what’s your length? Somewhere in the comment you see it’s length in millimeters, meters, or feet. Price usually means the price and the currency, the main currency of your organization. And that works well unless you have the software going internationally, so then you need to change that. And then you cannot simply add a currency field, because that would change the semantics of price. So there would be a breaking change because some, let’s say, implicitly was price in US dollars. Now you have a currency field and you have old client reading price and doesn’t know the currency, then it will interpret this field as dollars, which it might not be at all. So if you want to do this, you need to keep price, and then you define probably price with currency, and you need to support both of these elements, which makes your provider code much well, harder or more difficult to maintain over time, because you have all these different things where you have multiple ways of expressing the same logic or transferring the same data.

In most real-life cases, you just don’t want to guarantee endless backwards compatibility. And then the question is how do you manage that? Do you deprecate things? And deprecational tool would be one strategy for in-house applications and also for two in production things. You know, we have two API versions active and you can then migrate over and then the older one will be gone in an announced timeframe. Or you just say from the beginning, okay, I guarantee you that for the next two years, that’s stable, which would be a time-based guarantee. And then, obviously, you can just pretend to be backwards compatible forever, which will lead you probably down the way to the deprecation approach, anyway. Because at some point you will realize, okay, I need to do something breaking, and then I just announce how this works.

So, basically, these are the strategies and you can combine them, obviously, plus the experimental preview we talked about where you say, again, I’m not really in production, I just don’t give you any guarantees. And then you have covered everything and then you need to choose and pick one or a combination of these. But you should do this because that’s not a question of versioning. It’s a question of lifecycle. How you want to force the clients into certain life cycles?

Henry Suryawirawan: I think it’s pretty insightful. Because for those engineers who have developed their API for long, there’ll be multiple versions, multiple iterations of your APIs, and this is where it gets tricky, right? How do you support backward compatibility? How far do you want to support it? So you mentioned a couple of patterns here. Things like, for example, two in production, limited lifetime guarantee, which is like, how long do you want to support the API? And you also mentioned experimental preview earlier. And one more is about aggressive obsolescence. So tell us more about this aggressive obsolescence. Is it something like you force the client to actually just break?

Daniel Luebke: Yeah, actually that’s the deprecation approach. As long as it works, I stay backwards compatible and make it work for you, but I also have this mechanism of telling you, okay, we deprecate something, and from our provider perspective, we’ll try to do this in an aggressive way, because we want to reduce our technical debt. At some point, we’ll say, okay, this operation or this data element is deprecated, and you have, well, like say, two months or a year, depends on, obviously, your clients and the power the clients have over you or you have over your clients. You give a certain time span until they have to migrate. And if they don’t migrate, well, they will break because you will discontinue this feature. So that’s what this aggressive is about.

Obviously, we have other deprecation approaches. If we look at the Java class libraries, there are things in there which are deprecated, like 20 years or so, I don’t know. So they obviously say, okay, you shouldn’t use that. But we place a high emphasis on backwards compatibility, which costs them as well. From API provider perspective, you want, or you sometimes must, we talked about this currency and account numbers, and so, there you must make breaking changes. Sometimes you want to do breaking changes because you want to refactor things. That’s what’s placed under aggressive, you know, want to give a sensible timeframe, but you have a certain timeframe when you will definitely phase out some features.

[00:39:27] API Versioning

Henry Suryawirawan: A little bit of question about maintaining all these versions. First of all, always there’s this paradigm. How should we version it? Is it like version one, version two, version three? Or is there any other strategy things like semantic versioning and how do you actually test if you have multiple versions that you have as an API provider? Is there anything special that you have learned throughout your journey to test all these different permutations of APIs that you have?

Daniel Luebke: Yeah. You now will have a flame war on the comment sections. Yeah. You know that, just by placing this topic. Well, let’s say, there are two schools, one loves versioning or having some kind of version identifier visible in either your endpoints or your URLs or in the messages being transferred. Perhaps also in the content type if you are on HTTP. And some people say it’s just the most evil thing you can do.

From my experience, I like to have some version indication. However, it should be clear that you shouldn’t place any logic, so any business logic. So if it’s version two in this field, price means this. And if it’s version one, this field price is definitely a dollar, so that you shouldn’t do. But you should be able to signal some kind of compatibility. So let’s say you have a mobile app and the user hasn’t upgraded it or has an old operating system on its mobile that can’t use your new app version. You want a mechanism where you can say, probably this doesn’t work, and just say, okay, no, you can’t use this API if you are that old. Semantic versioning has the advantage of indicating what kind of changes you make. So if you have these three versions, major, minor patch or fix, depending on who tells you what this last digit is named. But it’s very clear if you have a major change, it’s breaking. And it’s clear that if you have a fixed version change, it’s just a minor cosmetic thing which shouldn’t break anything on the client side. I think that it’s worthwhile to see, so you have some possibility of estimating the impact of these changes.

Yeah, I think that’s more or less answers your question. So you have these two parties in the API community favoring or disfavoring explicit versioning. And if you version, many people use semantic versioning to indicate at least some kind of compatibility.

Henry Suryawirawan: And how about testing those different versions that you have? Do you especially need to provide test cases for all?

Daniel Luebke: Yeah. You should have test cases for all. What we did, just referring back to the Swiss project, because they’re very open and transparent and can very openly talk about what we’ve done there. So, their policy was you have maximum up to three versions. So, when a new version is released, might be three, and the oldest one will then be taken down two years later. You see, it’s a very not Agile environment, very high lead times. So it means you have possibly three banking API versions, three land registry API versions, and three notary API versions. The same process might be under nine different API version combinations. So that’s pretty bad.

Actually, what we did there was we generated test cases. So we lay out a way of specifying the data. So on a more business level, more specification level, and then we have generators all for creating messages, conforming to these different API versions. And, obviously, that’s one extreme side of things. Not in all projects you benefit from the generation, because generation itself also costs something. We have a different cost structure and so we have more upfront effort, and then things get easier. If you just have two APIs, let’s say you have version one and two. When you release B, you always have your test cases for A already there. When you adapt them to cover version B, and if you then some point in time, take down version A by whatever means, by deprecation or cause you have two in production or reach to the lifetime guarantee, then you will also do need the test cases.

And that also works for if you say we are keeping backwards compatibility. So going back to this currency example, and you have a test case for version A, which checks the price, is it working. So I don’t send the currency with the price correctly as US dollars. And I have a test case which sends a different currency, which would resemble version B. And so, it doesn’t mean that I need to explicitly version things, but I should also have test cases. I just evolve my API without version identifiers.

Henry Suryawirawan: So for people who are not producing API, as like they sell the APIs, right? Sometimes engineers tend to forget about this testing part. They always keep evolving for the new one, new versions, but they don’t realize that actually their old API or the older version of their API, there are still some people using it. And whenever we make these new changes, it will break. So I think what you mentioned definitely makes sense, right? So for people to always have test cases for all the APIs that you still support that is still live as part of your API provider.

[00:44:27] 3 Tech Lead Wisdom

Henry Suryawirawan: So Daniel, it’s been a pleasant conversation. There are so many things wrapped up in the book. I think it’s pretty thick the way that I see it. So many patterns, starting from the basic one, the fundamental. What is API? What is the structure, protocol, and things like that. Up to the patterns. I think for people who want to build better APIs, I would definitely recommend reading Daniel’s book. But, unfortunately, we need to wrap up this episode. But before I let you go, I have one last question that I normally ask for all my guests, which is for you to share what I call three technical leadership wisdom. You can think of it like an advice that you want to impart for the listeners here to learn from your experience or journey.

Daniel Luebke: Yeah. So I think first is lead by competence. So if you are a tech lead, by definition, you’re leading a team of junior and perhaps senior people, and you have people who are trying to say, okay, I’m higher in the hierarchy, so therefore, my argument wins. That’s not true. In multiple ways, it’s bad behavior. So one is you are not really respected by your team. So you are respected usually in the development community, because you are good at something. So that’s one thing. So if you can persuade by bringing good designs and good suggestions, and you are able to say why they are good. Not, “Well, I’ve done this for 20 years”. So that’s a bad explanation. But if you can say, okay, we should do this here, because otherwise our database load would be too high, for example. It’s much better. So lead by competence. The other side of the coin is you also learn. There’re so many new things going on there, and junior developers are very often much better in knowing the newest framework versions, and so deep in the technical details. They’re not as good in conceptual thinking, hopefully. But you can still learn things. Should be open to learn as well. And if you just say, well, I’m the boss here, then you won’t learn.

Number two is, keep on thinking. Well, people often just do not think enough about their decisions and why they make this decision. So we have like architectural decision records where we can document why we do something. We can say, okay, patterns. Make explicit what we want to achieve and what we lose. So really making conscious decisions about things and not just going along.

Yeah, well, the third is, it’s also hard for me to translate, but it seems probably you will laugh, but anyway, so make the world a better place. So, within your project, within your team, within your company, and probably with the product you release. It’s so incredible what you can do with software. What we can really achieve in this world, like I said, like in our team, up to when things are released. So imagine you are an engineer in Tesla, and at some point in time, you have the greatest autopilot and cars are moving by themselves. So how great is that? Obviously, there are more boring projects, but still even the Swiss Land Registry thing is. We now moved back to Germany. We bought a house last year. And it was all on paper. Because of my software, there’s no paperwork, and here now I’ve got piles of paperwork. And then I realized, well, it was not just a boring administrative product. That really made people’s life better, and that’s very satisfying, I think.

Henry Suryawirawan: Wow. That’s a very beautiful message. So make the world a better place. So yeah, a lot of engineers, when they do work, sometimes they don’t think about what impact or what values they bring to the larger society. Not just for the company, but to the human, the people, the customers who use the product. So I think that’s a pretty good reminder for us to actually know and align what exactly the value we bring to the users and the customers out there. And best if we could bring it to the world, right? I think that’s pretty impactful.

So Daniel, if people want to connect with you or learn more from you, is there a place where they can find you online?

Daniel Luebke: Yeah, sure. So it’s on LinkedIn, Daniel Luebke, so I know it’s a very international first name and the pain for all non-German speakers’ last name. So if you search within “ue”, you’ll find me on LinkedIn and on Twitter, just dluebke. If you want to read my blog, it’s on digital-solution- architecture.com.

Henry Suryawirawan: I’ll put those in the show notes. So thank you so much for your time today. I really learn a lot about API design pattern, and hopefully all the listeners here also learn a lot from you. So thanks again for that.

Daniel Luebke: Yeah. Thank you for having me. It was a pleasant discussion.

– End –