#64 - Principles for Designing Successful Web APIs - James Higginbotham

[00:00:44] Episode Introduction

[00:00:44] Henry Suryawirawan: Hello to all of you my listeners! I’m happy to be back here again with another new episode of the Tech Lead Journal podcast. Thank you for spending your time with me today, listening to this episode. If this is your first time listening to Tech Lead Journal, I would invite you to subscribe and follow this show on your podcast app and social media on LinkedIn, Twitter, and Instagram. And if you have been regularly listening and enjoying this podcast and love the type of content that I’m producing, will you support the show by subscribing as a patron at techleadjournal.dev/patron, and support my journey to produce great episode every week.

Building web APIs has become ubiquitous whenever we build tech products. Almost everything is nowadays available on the internet. And it’s just one API call away to integrate disparate software systems in order to make them work together to perform a series of tasks or activities. Think about your mobile apps, software as a service, electronic payments, or even smart home devices that you use frequently. They are all connected seamlessly through APIs that power the integration. And the importance of the APIs is not all about choosing the underlying technologies or protocols, such as whether we should choose REST API, GraphQL, gRPC, and so on. There is something more beyond that.

My guest for today’s episode is James Higginbotham. James is the author of “Principles of Web API Design” and an executive API consultant. In this episode, James explained why it is extremely important to design APIs properly and shared the five key important principles of API design, such as not to design it in isolation, importance of API documentation, etc. James also recommended the API design first approach, which is a rapid and lightweight outcome-based API design process to design and deliver APIs successfully, including the ADDR process (Align-Define-Design-Refine), establishing API boundaries and how it relates to Domain Driven Design or DDD. Towards the end, James also shared some recommendation for API testing strategies and also some API testing anti-patterns that we all should avoid.

I really enjoyed my conversation with James, understanding the importance of a good design principles for building successful APIs, including his design first approach and the ADDR process. And if you also find this episode useful, please share it to someone you know, who would also benefit from this. And you can also leave a rating or review on your podcast app and share comments on the social media on what you enjoy from this episode. I always love reading your reviews and comments, and they are one of the best ways to help me spread this podcast to more people. And it is my hope that they can also benefit from this podcast. So let’s get our episode started right after our sponsor message.

[00:04:15] Introduction

[00:04:15] Henry Suryawirawan: Hello, everyone. Welcome back to another episode of the Tech Lead Journal podcast. Today I have with me a guest named James Higginbotham. He’s actually an expert in API design. He has been a software developer and architect for more than 25 years. So that’s really a long time and mostly helping people to develop, deploy APIs, apps, and also helping companies even to go through transformations based on these API. So James just also published a book titled “Principles of Web API Design”. It’s something that you can look for when you need to build a web API for your applications, for your teams, or for your company. All these will be covered in this episode. So thank you so much for coming to the show, James. Looking forward for this conversation.

[00:04:57] James Higginbotham: Yeah. Thanks, Henry. I appreciate you having me. I look forward to our discussion.

[00:05:00] Career Journey

[00:05:00] Henry Suryawirawan: So James, maybe in the beginning, you can introduce yourself for those who haven’t known you yet. And then maybe you can highlight turning points or major highlights in your career.

[00:05:09] James Higginbotham: Yeah. So, for those that are not familiar, I’ve been consulting for a number of years ranging from startups and Software as a Service companies to enterprise IT. A lot of what I do is helping organizations to establish, grow, and mature their API programs as part of a larger digital transformation initiative. Part of that is coming in and working in a one-on-one or team basis to help organizations to grow their expertise, and that also oftentimes includes training. So doing live in person or remote workshops on API design techniques.

Interestingly enough, I got started in software quite a few years ago. You’re talking about turning points in the career. My grandfather gifted me a Commodore 64 computer when I was about eight years old. He said, “I want to get you this because I think computers are going to be big someday, and I’d like for my grandson to learn how to use them.” I took to it and turned it into a career. It’s been an absolute, amazing opportunity to go from kind of the early client server days, through client networking, where we were figuring out how a network systems across either a private network or over the internet, and then eventually into web development. And that’s also led into web API design, and helping organizations think about how to treat their web APIs not just as a technical asset, but really as a product that’s owned and matures overtime.

[00:06:33] Henry Suryawirawan: Thanks for sharing your story. Interesting enough, this is not the first time a guest actually shared about Commodore. So it seems like that changed a lot of people’s career. I personally haven’t seen it, so looking forward one day maybe to see the real product of Commodore, right?

[00:06:46] Principles of Web API Design Book

[00:06:46] Henry Suryawirawan: Let’s talk about your book itself, “Principles of Web API Design”. What do you think are some of the important things that led you to writing this book?

[00:06:55] James Higginbotham: A lot of this book comes from practical application that I’ve spent with organizations, and myself personally, designing APIs. I come from kind of a blend, as I mentioned, in my career of sort of the older school way of doing things where we used to build, or design and build everything up front which we don’t want to do. We learned other techniques for that. But I’ve also seen a lot of opportunity to think about API design in new and interesting ways. I’ve always referenced an old partner and co-author of mine, Keith Casey, who helped me put together a self-published version of what turned into this book when we put it together years ago. He always said that API design represents the effort, required design the third user interface, or we referred to an API design as the third user interface. It’s the UI for developers. What I’ve really taken from that is, API design centers on effective communication. And it’s not just communication between developers, but it’s communication that combines product thinking, business and technology all in one, and so it’s always been a real interest to me. So getting an opportunity to put a full book together after having years in the field, working with teams, coaching them API design and different techniques that you can use to get there effectively and quickly without bogging down the delivery process was always been a passion of mine. So that’s been the focus of this book.

[00:08:17] Importance of Designing API Properly

[00:08:17] Henry Suryawirawan: So, first of all, why is it important to actually design an API? So you mentioned that your co-author of the book told you about it is actually one-third of the user interface. Why is it so important to design API properly?

[00:08:31] James Higginbotham: I think it really does go to communication. We don’t oftentimes think of our API design as part of the communication effort. We think of the documentation, the web portals, developer portals, marketing materials and other things. But our design itself communicates. The words that we choose, the patterns that we apply, they all communicate to us. And I think as developers, we all encounter or have encountered web APIs or even just helper libraries in our favorite programming languages that were either a pleasure to use, or really difficult to use and really frustrated us.

So I think taking the time to consider your API design, and centering it on an outside-in thinking process, thinking about the outcomes that are desired, and that we’re trying to deliver with that API, really contributes to that effective communication, and also enables great documentation as a result. Because it makes it easier to document the API and communicate how to do things. You don’t spend as much time saying, “Here are the little nuances or caveats or things that are kind of non-standard that I put in here.” Instead, the API design becomes very intuitive for the person trying to use it to build a solution, to solve a problem, to automate something. So I think there’s a lot of importance to be placed, and effort centered around API design first.

[00:09:49] Henry Suryawirawan: And I must also assume when you say about all these communications, it didn’t just happen for a lot of people who build APIs. Maybe also based on your consulting or expertise seeing people doing this stuff. So why do you think those people do not actually emphasize on getting the API communicate it with so many people who are connecting to it?

[00:10:10] James Higginbotham: I think for years, API design was really pushed to the technology teams. Over the years, we’ve had to make decisions about frameworks and helper libraries, even entire programming languages, platforms, cloud infrastructure. Anything along the way, a lot of those decisions tend to center around technologists. But as our organizations start to think about how we expose web APIs to allow others to integrate with us, whether it’s Corp to Corp, just business-to-business kind of integrations, or if it’s more of allowing consumers to access thrown data or to automate solutions. Whatever the case is, we really have to spend more time emphasizing the API design. So I think up to this point, it’s just been kind of pushed off to the side, and now we’re really seeing not only tech leadership, software architects, senior leaders, any developer on a team now responsible for producing, delivering an API, either for internal use or maybe external use. But we’re also seeing product owners get involved as well.

So now, what we have is this opportunity to really think about APIs as products. And that is requiring us to take a step back, and to rethink the approaches we have to the way we approach APIs from the design perspective, from the coding perspective. Slow down, and meet those parties that have input, subject matter expertise to bring them along with us in the API design, and not just to forge ahead with the code. But rather to take the opportunity to turn it into something that centers around more product-oriented experience.

[00:11:46] Henry Suryawirawan: So you mentioned that product owners sometimes get to be involved. But is this only for the case where you are building an API as a product? Or is it also applicable for people who build like backend APIs where you just connect from your mobile applications, or maybe your web front ends? So any thought about that?

[00:12:04] James Higginbotham: We’re starting to see more and more product owners get involved, whether this is an API for a Software as a Service, an API as a product offering that an organization is putting together and, releasing, or if it’s just internally facing. It’s still taking a bit of time because I think from the product owner perspective, their responsibilities and what they’re held accountable for tends to still be related to either a partner-to-partner integration or a web or mobile app, so we tend to still focus on the wire frames and the features and the function of the application. But it’s starting to become more and more apparent that through these digital transformation efforts, anything we do, whether it’s meant to have a human interface in some way, voice, web, mobile, whatever it is, or whether it’s automation, that there’s an opportunity to consider the API as part of the product offering. And so that’s resulting in a lot of product owners starting to go beyond their initial focus of just the product as it’s seen and experienced, and more of the holistic view from front to back. And that includes the APIs that help to power those solutions as well.

[00:13:13] Principle 1: API Should Never be Designed in Isolation

[00:13:13] Henry Suryawirawan: So let’s go through some principles that you actually mentioned in the book. The first principle is that you said, “API should never be designed in isolation”. What do you mean by that?

[00:13:23] James Higginbotham: That’s really what I’ve been talking about here, and that’s why I’ve spent a good deal of our interview so far discussing it. Because we have oftentimes required our technical teams, our delivery teams, to shape the APIs themselves. While there are quite a few developers out there that have designed well thought out APIs before, I think there’s an opportunity to expand our influence, and to incorporate other parts of our business in the process. So in this case, we’re going to have, as I said, subject matter experts from product owners to perhaps even security teams or the office of the CISO that is looking at the security aspects of an application and what kind of security concerns there are with an API. You know, personally identifiable information, and non-public information, other bits of information we may not want leaked. There are different subject matter experts that should be consulted when we think about the APIs. Too often, we’ve released web APIs out, and either not secured them or didn’t think about the design from those different perspectives from security, from product ownership, from the user experience and developer experience.

So the first and foundational principle is we should never be designing APIs in isolation. We should be looking for those other team members to come with us. We also have to kind of slow down as developers, and if necessary, educate team members that may not be as familiar with some of the technical details of HTTP and other things. Just kind of help meet them where they’re at, and partner up with them, and move forward with the API design with them involved rather than just going off in a corner and building something. Cause there’s a lot of opportunity to make APIs better, to make them more secure, and to make them longer lasting.

[00:15:10] Principle 2: Outcome-Based Focus

[00:15:10] Henry Suryawirawan: Yeah, which I think also led to the second principle, which is you mentioned about “the design should start with an outcome-based focus”, right? So when you gather all these people around, I’m sure there’s a common outcome that you want to get out from this APIs. Is that correct?

[00:15:23] James Higginbotham: There is. What I mean by the outcome-based focus is not just what these team members are including or wanting to factor into the API design. But there’s also the aspect of who’s going to be using the API? And what they’re trying to do? So part of what I do is coach on a technique called the jobs to be done, which started with Clayton Christensen who wrote “Crossing the Chasm” and several other business books. His perspective was whether you’re building a startup or whether you have a large global organization. Anything that you do, service or product, is solving a job to be done. And the best way to be successful at it, and find something that is going to do well, is to understand the customer, listen to the voice of the customer, understand their problems and what their desired outcomes are. If we take those things into consideration, then the piece in the middle is the job to be done. If we understand what the situation is, and what they really want the situation to be, the end-state, then what we put in the middle is that job to be done.

So when we think about our API design, we want to think about the same thing. There’s a tool called job stories, which are kind of similar to user stories, but they start with the “When”, which is that situation, the “I Want To”, which is the job to be done, and the “So I Can”, the outcome that they desire. When I talk to teams about API design or when I’m designing my own APIs, that’s where I like to start. That means that all those team members that I mentioned from principle number one, that are subject matter experts and have input to provide, are going to be contributing to that understanding. And that might require our product owners to take a step back and start talking to the target audience. Get the feedback just like we would any other product, and use that to help drive what our API design needs to deliver, and look at what kind of outcomes we can produce as a result of using that API, either as a standalone or in collaboration with others.

[00:17:17] Henry Suryawirawan: It’s very interesting you use this concept that is normally used for building products. So like when we want to gather user stories or requirements, we use this jobs to be done framework. But you mentioned that you actually use this to actually come up with the API design, what you call job story. You mentioned about When, I Want To, So I Can. So it’s really interesting that you actually bring all these best practices from product development or product specification, to actually web API specification as well.

[00:17:44] Principle 3: Design Elements That Matches the Needs

[00:17:44] Henry Suryawirawan: The next one in your principle is actually, you mentioned that you need to select API design elements that match the needs. So I’m not very sure about this principle. Maybe you can help to explain about it.

[00:17:55] James Higginbotham: Yeah. I think this really stems from the idea that there’s been a lot of discussion over the years. Do I use a Rest API style? Do I use GraphQL? What about gRPC? I’ve shared discussions in the past and spoken at conferences where I kind of dispel the myth that there’s any one particular API style that’s best. So when we think about the design elements of our API, we need to think about how people are going to be using the API. What’s the context in which they’re going to be using it? Is it primarily from a web browsing device, a phone, tablet, laptop, whatnot? Is it a voice-based? Is it that we’re going to be integrating with other systems, and so we need some events? So we might think of things like webhooks or server-sent events or Websockets, maybe Kafka Streams if you’re sharing messages within an organization across teams across systems. All those things need to come in together to match the need of the consumer. So as we’re taking a product-based approach, part of what we need to be thinking about is not only what the API design should look like, but how, or what API style or styles, more than one style, should we apply to make it the best match for our consumers?

So just because an API producer, a team that’s designing and building an API, fell in love with GraphQL, love it. If their audience that’s going to be using API is not familiar with it, they have two choices. One, they proceed with GraphQL, and they invest in educating the audience on how to transition to GraphQL for those that may be more familiar with REST or other styles. Or two, set aside their desire as a provider and as a single team, and think about holistically, all of the different teams that are going to be consuming this API, and determine if a different style would be a better match. Or maybe it’s offering choice. Offering both REST and GraphQL, and maybe even some event capabilities with webhooks for callbacks when event notification and reactive style of interaction make more sense.

So it’s just the idea that we want to recognize that there’s no one single solution. That there’s no one thing that is best. But if we factor in our target audience, then we’ll be able to determine what the best fit is at least for today, and we can continue to listen to them, and then offer new styles as the needs arise. That keeps us from being a little too focused on ourselves and a little bit more focused on the variety of developers around the world that may be wanting consumer API, and would prefer to do it in the way that they would like, rather than the way that you might want to mandate.

[00:20:34] Henry Suryawirawan: I know that you mentioned there’s no best API styles of API technologies, but for those techies who listened to this, I’m sure they will have the same question. Is it REST? Is it gRPC? Or is it GraphQL, right? The new darling in the API world. But from your view, maybe you can give us some summary. When do you think we should choose each of those popular technologies?

[00:20:54] James Higginbotham: I found that REST or just basing a web API on HTTP, typically using the REST architectural style, is a great approach because the tooling exists in so many languages. It’s so easy. I can use a variety of developer tools to assist me in the development process and the design processes in consuming the API and trying things out. I just see that as a popular option for a lot of organizations. That said, I’m seeing some organizations starting to go down and explore GraphQL a little bit. They’re finding that it’s a really great fit for the response shaping aspect of it, even though there’re other aspects, that tend to be where people gravitate toward. The idea that I can craft a query that looks a lot like a database query, and I can be very specific about the elements that I want.

So if you have a very deep resource structure where you have lots of nested resources, you don’t want to have to do query upon query to go deeper and deeper, and you don’t want to implement your own version of that on top of REST, oftentimes, the teams will choose to use REST as a foundation complemented with some GraphQL. Then maybe if they need some high performance service to service communication, they’ll decide to go gRPC. Speeds up the development process with the code generation that’s built in, and also with some of the performance improvements the gRPC takes advantages of from HTTP/2. Some of the compression of headers and other things that are built into it, and the bi-directional communication, it gets teams a lot further. So again, it’s not a one size fits all, but I do see the majority of APIs that I work with tend to start with REST, and then they look at other styles to complement it. And then we see things like Websockets and webhooks used for eventing quite frequently.

[00:22:36] Henry Suryawirawan: And I think you mentioned also when you explain about it, you can mix and match. So not only choose just one style, but you can actually start with providing REST and maybe some gRPC on top of it, or maybe GraphQL for those people who really want to shape the response of the web API. So thanks for sharing that summary.

[00:22:53] Principle 4: API Documentation as UI for Developers

[00:22:53] Henry Suryawirawan: So let’s move on to the next principle which is I think a very, very interesting because you mentioned also in the beginning that API documentation is probably the most important user interface for developers. I’ve seen in my whole career, API documentation seems to vary. Some are really good. Some even have the capability for you to test it on the documentation itself. But some are just like, okay, what’s the endpoints are, the parameters and all that. So maybe you can explain a little bit. Why do you think the documentation is the most important UI? You mentioned it as a UI for developers.

[00:23:24] James Higginbotham: The reason being is there’re different paths that we all take to work with an API. So as a developer that’s looking at a brand new API, the reference documentation is very important. I think all of us know whether it’s a helper library for our favorite programming language, or whether it’s a web API that’s allowing us to take advantage of a Software as a Service solution, or whatever it is, the reference documentation is where we spend the bulk of our time. However, there’s additional documentation on top that’s really important. Getting started guides that help frame up and position the API as to what it does, what it’s capable of doing. Cookbooks or step-by-step guides that show you how to do common tasks are really important as well. So we sometimes forget, particularly when we’re designing our own API, that just because we’ve spent a lot of time thinking about it, designing it, implementing it, deploying it, troubleshooting, getting it ready and pushing into production, and we’ve managed it in production for some time and we have people successfully using it doesn’t necessarily mean that next developer that encounters your API is going to know exactly what your API is meant to do. What you’ve designed it for? What it’s good at? Maybe what it doesn’t focus on? Or that you would recommend complementary APIs to pick up where you leave off. And how to get started?

Those types of aspects cannot be under-emphasized when we think about an API. And the last thing that we want to do as developers is document. I don’t know why. A lot of us have developed this habit that done is when the code is done and the tests are passing, and I can get a pushed out to a cloud resource and it’s up and running. But if you don’t document it, then those developers that are trying to understand how to use it are getting very frustrated. Just like you probably get frustrated with a particular website that’s poorly designed, or you know, a word processor or a spreadsheet that’s just not super intuitive or a mail app that just doesn’t work the way you want it to work. I think all of us have probably used over the last 18-24 months, a variety of video chat and collaborative messaging platforms from Slack to Teams to WebEx to everything else that’s out there. And we have our preferences for what we like to use and what we don’t.

That user experience or that frustration that we have is oftentimes for web API is rooted in great or terrible lacking API documentation. So taking the time to document, even if it means asking for outside help, getting a technical writer that’s really good at asking you the right questions and getting the copy so that people can understand it and it’s approachable, will make your API very popular. Because it stops the frustration cycle, and it helps people get things done. Some of the most popular web APIs we know of, Software as a Service APIs that we know of, that’s what they did. So even if you’re building an internal API, you want other people to use your API internally. Write a little bit of documentation. And if you’re not setting up a whole portal, just take the OpenAPI Spec or whatever it is you’re using to capture your documentation. Spend a little time, put a few examples or walk people through some basic use cases or usage examples. Help people get started. Have some empathy for the people that are seeing your API for the first time. Maybe they’re having a bad day. Now they have to go get something done, and this API is going to help them. Give them a good day, not a bad day. So spend some time documenting or get some help if you need to. Get an assist from somebody, contract somebody in, get somebody else in the organization that might be better at it to give you some tips. Whatever it is, and really focus on the documentation.

It can occur over time. It doesn’t have to all be done immediately. But over time, you should be improving it. So start simple, grow it, improve it. Don’t let it be the last thing on your to-do list when you release a new version of the API or make an improvement. Constantly look for ways to improving it. Take feedback from troubleshooting sessions or support sessions with developers trying to use your API. Take what they were struggling with back and try to improve the documentation and just continue to do that. It’ll go a long way to making a successful web API.

[00:27:23] Henry Suryawirawan: Yeah. So from my experience as well, it’s not just the tooling actually to build this API documentation becoming more varied and becomes easier as well. So like you mentioned OpenAPI Specification, what used to be known Swagger, I think, and also tools like Postman. So they evolve into building ecosystem or building tools for people collaborating building API or using other people’s API. So I think, look for tools as well. Not just like building by yourself. I think that’s also key here.

[00:27:51] James Higginbotham: Absolutely.

[00:27:52] Principle 5: APIs Are Forever

[00:27:52] Henry Suryawirawan: Yeah. The last principle that you’ve mentioned in the book is actually, this is very interesting, definitely. You mentioned APIs are forever. It tends to stick, so you need to plan accordingly.

[00:28:01] James Higginbotham: Yeah. I think many of us in the API space have dealt with web APIs can understand this one. And I know that Werner Vogels from his days at AWS really kind of solidified this one through various posts that he’s made over the years regarding the challenges and the opportunities and the successes that AWS has had with their APIs. That has really codified the idea that APIs are forever. I mean, if you think of the S3 API, from AWS, just that API alone. Imagine if you woke up tomorrow, and they broke something, they changed an API, a signature, they renamed something, they dropped an operation that you needed and that you were dependent on. You woke up and things just stop working. We just can’t do that. When we think about web APIs, we can’t just think about a code base that we can refactor and change things with. And as long as it all starts to compile and work again, we’re okay. We’re talking about distributed systems, many systems of which are outside the control of other people using it. So those that are consuming your web API are out of control of what you do as the provider and vice versa.

In that case, we have to have a strategy for how we manage versioning. How we approach it? Being sure that we’re not breaking people, and also making sure that we’re not creating brand new versions of APIs every week or every month. It’s great to enhance an API with non-breaking changes, you know, additive changes. Those types of things are great. But removing things, renaming things, those things that would break someone who’s using that operation, who were using that particular field in a response. Or using a particular feature that this part of your web API, and you just decide one day, I’m tired of maintaining it, or I’m tired of the way I named this, I’m going to refactor the code. That refactoring ends up changing your web API because you haven’t separated the web API contract from your implementation details, and so you’re unable to make it still work the way it was designed, even though you’ve changed things internally. If you start to introduce breaking changes, then you’re going to introduce opportunities for customer churn. So, if you’re as a Software as a Service, you have a web API and people are integrating with your web API to automate or integrate your Software as a Service capabilities with other things. That they use other Software as a Services, and they’re doing things to help them and automate their lives, and you make a change. They now have to go make a code fix. Well, they now have an opportunity, or a question, a choice ahead of them. Do I make the code change? Or do I go find someone else who treats customers that are integrating with our web API with more respect, and prevent breaking changes from coming in, and we waking up to a bunch of problems.

So the way that we treat our APIs. The way we version them. The way we manage change is really important, and we have to plan for that. There’re strategies that I outlined in the book for that, and different techniques you can think about that afford you opportunities to make breaking changes before you’re committed to those designs. Before these designs become frozen where you can only add new things, but you can’t go back and change those things you’ve already put out there. There’re ways to do it. But keeping that in mind is absolutely important. Too often we just plunge headlong into the code. We think we’ve got it all figured out. We put it out there. It doesn’t work, or it doesn’t meet the needs or expectations. Yet people start using the API in different ways, and then we want to break it to introduce the right way of doing things, because we don’t like the way it turned out. We end up causing too many problems, and we want to avoid that.

[00:31:19] Henry Suryawirawan: So I, myself personally have experienced, for example, there’s an API deprecation announcement or emails, “You have to move to the new API by this”. So personally, I find sometimes it’s very annoying because what is working well in my code base, suddenly you have to change because of this. I think it’s very important that you have this mindset that API that you design and publish tends to stay forever, right? Don’t change it too often.

[00:31:43] API Design First Approach

[00:31:43] Henry Suryawirawan: Which brings us to a good segue to the next discussion. You mentioned API design first approach. For many, this doesn’t come intuitive. Because normally they will just start from product requirements, just code and showcase. Especially when you integrate with third party between services. So you mentioned here the important concept that I learned throughout my career as well, that API design first approach tends to be the best practice here. So maybe you can elaborate on why do you think it’s the most important thing?

[00:32:10] James Higginbotham: I think when we think about building software, we went from sitting down and reasoning about a design and spending a little time designing out our software. Now granted, a few decades ago, we spent too much time there and not enough time actually producing or delivering. I think the pendulum swung the other way, where we said, well, code is most important thing. Our code is our documentation. We can just kind of evolve our design as we go and figure things out. Yeah, we can do that in small bursts, in small increments inside our code base. Many of you listening may have experienced success in that way, and I don’t discount that. But you also have to recognize as we said with all these different principles that we can’t design in isolation, that we have to think outcome-based, that we have to think about how our consumers want to interact with our APIs, how we’re going to document them. And that once they’re out there and we have the first integration with our API, it’s out there forever. We don’t want to be tasked with having to keep up with your API changes because you keep changing your mind on how your design should look, and I’ve got to keep up with it because you’re deprecating a version and forcing me to write more code and adjust my code to make things work.

So there needs to be a balance between what we do with upfront design and how we deliver. The book outlines an API design first approach. The idea that we need to spend a little thought and incorporate this input from other team members, and to think about an outcome based focus of what our API is going to do. If we spend a little time doing that, then we don’t have to experience what many of us who’ve experienced, and I know I have, is writing code that never makes it into production. Because we completely missed it. We made too many assumptions that were incorrect. Or we jumped too fast to write the code, and we didn’t ask any clarifying questions. So what we delivered missed the mark, and now we have a few days to a couple of weeks to try to make something work, to fix the things we need to fix. And so we then start saying, well, that’s going back on the technical backlog. Well, that’s going in the backlog, or I’ll fix it on the next version. I’ve experienced that way too many times, and it was frustrating. So finding that balance with web APIs in particular, because they are forever.

Principle number five, APIs are forever. To think about how do we do a lightweight, rapid API design first approach that will get us closer to the right mark. Prevent us from writing or designing an API that’s incorrectly or misaligned with what really is needed, and still get feedback quickly so that we’re not doing this totally in isolation in our own corners, and it’s too costly or too late to make those changes. So that’s what the Align-Define-Design-Refine or ADDR process is all about.

[00:34:43] ADDR Process

[00:34:43] Henry Suryawirawan: Yeah. So maybe you can go and explain what is this ADDR process?

[00:34:48] James Higginbotham: It’s Align-Define-Design-Refine. The Align is how do we ensure that the API that we design is aligned with the outcomes that are desired by the developers who are going to be using the API, by the end users that will be indirectly using that API, maybe through a web or mobile app or anything else. Gaining alignment by starting with the job stories, detailing those jobs stories out to the next level where we have step-by-step. We understand the stepwise processes involved to achieve the outcomes that we found in the job stories. And then defining our API by looking at those, finding our API boundaries, and the API operations we’re going to need. Independent of any one particular API style. So I’d mentioned earlier that, you know, we can mix and match things, REST and GraphQL and gRPC. So if we align, first and foremost, to make sure that all of our assumptions are dealt with, that we’re not writing code that has assumptions built in that are incorrect. We’re not designing an API likewise that has incorrect assumptions. We understand what the needs are, what the outcomes are. We break those down to the next level and understand step by step what we need to do. Then the API design will evolve along with it.

So we go through the Align phase. We go into Define where we turn that step-by-step process into APIs that have operations to fulfill those step-by-step processes. That produces those outcomes we found in the Align phase. And then we apply one or more API styles to design the API. So we might apply a REST approach for the majority of our operations. Maybe offer a GraphQL. All using the output of the Define phase. An API profile that defines what the API is supposed to do at a high level. Protocol agnostic. And then once we’ve designed it, now we have this high level design that we can quickly push into an OpenAPI Spec, or GraphQL SDL, Schema Definition Language, or a gRPC IDL, Interface Definition Language. We can capture it and then socialize it. We can use those artifacts to socialize it. We can generate mock implementations of our API using tools that can take in one of those definitions and create like an in-memory mock implementation of it. Just so that you can kind of try it out without it really being completely coded up yet. That allows us to refine the design, and go back and rework that.

So the Align-Define-Design-Refine is the four major phases we go through. There’re different steps outlined in the book that take you step-by-step. How to go through the Align phase and make sure that everybody’s in alignment from business to product owners to the tech teams. And then how to define the API. Find those boundaries, find those operations, then apply the API styles you want to create the design. And then to circle around, gain feedback and do that in a rapid fashion. So, you spend a little bit more time upfront, but you save a lot of time on the back-end where it’s more costly to change code. And it can be very frustrating to have to take some well-written code, well-designed code, and make some last minute changes and really mess it all up because we misunderstood something in the beginning. It tries to encourage communication throughout. It’s really applying the Agile Manifesto principles right into this design process without being too front loaded or too heavyweight. So that’s what the book outlines, and that’s what I really recommend. And it allows us to have that design first approach. So we can talk about the design, look at how we would integrate with it and make sure it’s going to hit the mark and we haven’t missed anything. And then we can go write our code and do what we do best there in the different delivery phases.

[00:38:18] Henry Suryawirawan: So I also find this kind of approach very useful whenever you build both APIs independently, and also starting from scratch, for example. So especially if the team are dispersed geographically, for example, different time zones, but you need to collaborate through the APIs for those services. Having gone through all these Align-Define-Design, even providing mocks testing for people to integrate with the API, so it speed up the development so that they can all do in parallel. At certain point, you test it in a staging or maybe some kind of a testing phase where you can ensure that, okay, actually, these two services can talk to each other. Thanks for sharing this ADDR concept. I think it’s really useful.

[00:38:56] API Boundaries and DDD

[00:38:56] Henry Suryawirawan: Another thing I pick up when you mentioned about this ADDR is you mentioned about API boundaries. What do you mean by API boundaries? Is it something that is related to bounded context and all this DDD?

[00:39:07] James Higginbotham: It is. Yeah. So, people that look at the ADDR process and the techniques that we train on will recognize some of them from the DDD space. What I wanted to do, and what I have tried to prevent, is isolating those that are not familiar with DDD, or that are only practicing some of the techniques and preventing them from being able to take advantage of this process. So it is designed to be able to incorporate many of the concepts of DDD. We use EventStorming, encourage EventStorming during the Align phase to try to surface things. If it’s a domain that’s a little bit newer for some people, bringing in subject matter experts from the domain, and having collaborative sessions to explore and understand it, is really valuable.

As well as what you mentioned with the API boundaries. The idea is that I’ve seen a few APIs over the years that may have hundreds and hundreds of operations. Those are just so unwieldy to understand and to get started with. So unless you just really surround all of that really large API with a lot of copious documentation that guides the reader to where they need to be. Unless you do that, then it’s really hard to get started, and it’s really not approachable to have to take on an API that’s really large. So the idea is to bring in boundaries where we can segment parts of the API. That doesn’t necessarily mean that you have more than one API product. It means that you’re finding different APIs or different operations that are cohesive.

So I go back to the fundamentals of software development. High cohesion, loose coupling. Loose coupling says we don’t want to tie into the internals of our systems, into the implementation, very much like encapsulation. We want to hide the internal details from the external API. We want to loosely couple and just coupled to the API itself, not to the internal implementation details. High cohesion is about having related functionality grouped together in the same code base, rather than scattered all over the code base, where it’s hopping between module to module, what we call spaghetti code or a big ball of mud, where things are just all over the place and there’s no rhyme or reason for it. All those techniques that exist in DDD are really beneficial. Whether the team is early in their DDD journey, whether they’re not on a journey at all, and they just want to design really great software, or whether they’re DDD experts. Taking the time to evaluate where those boundaries are for your APIs will enable you to be more efficient and decompose a very largely scoped product or surface area of API operations into smaller bounded areas that are more digestible and easier to manage, and easier to decompose for delivery by the API providers. So we bring up some of those DDD concepts in throughout the process. But if you’re not practicing DDD, no need to be concerned, it doesn’t require DDD for you to be able to take advantage of it.

[00:41:47] Henry Suryawirawan: Maybe for those people who are interested in the DDD, if we can go just slightly deeper. Is API resource, like the term resource from REST here correspond very closely with the entities?

[00:41:58] James Higginbotham: They do. What we typically do, though, is we try to get teams to look at the aggregates. The aggregates tend to be where the operations of the commands and queries that you send in tend to cluster around. They tend to focus more on transactional boundaries and the collaboration of multiple entities together. So the resources typically represent the entities, but the APIs tend to scope to the aggregate. So we use EventStorming as a great way to find that we can lay out a particular process, start with our events and then start to expand and find the commands that generate those events, and then identify aggregates, and detail out the event storming process. Oftentimes those aggregates are really good hints of where API boundaries are at. We can look at what we’ve discovered and then reassess. So we can always make some changes. So that we have better cohesion if we’ve identified multiple aggregates, but maybe they belong together in a web API.

Because we have to keep in mind a lot of web APIs because they’re going over a network must be more coarse-grained than our DDD code bases that tend to be more granular and can operate in-process. So when you go to a distributed architecture where we have web APIs, we don’t want to have too much network traffic. We don’t want to be too chatty. So we have to start looking at how are we interacting over the network, which is different architectural style than how we might architect a standalone code base and process that may expose that API and how we code it up internally. Looking at DDD, we really want to focus first on the aggregates, and then allow that aggregate to help drive the API operations. And then behind the scenes, we continue to apply DDD at smaller and smaller levels as we realize the API during delivery.

[00:43:43] Henry Suryawirawan: Thanks for explaining all this and clarifying how do these two concepts actually help each other, or align with each other. So, thanks for sharing that.

[00:43:51] Testing APIs

[00:43:51] Henry Suryawirawan: Maybe the last point about API. So you kind of like design, you kind of like develop it. You kind of like put the documentation. But how about the testing strategies? Because at the end of the day, people just want to integrate with your API. First of course, they will need to go through the testing strategies. Maybe you can elaborate some of the testing strategies that you commonly see throughout your consulting?

[00:44:10] James Higginbotham: Yeah. Yeah. One of the anti-patterns that I see is that teams just starting off with building APIs because they’re oftentimes have a front-end to them. They’ll use their UI testing suites, whether it’s Selenium or something like it to drive front-end test suites to build those, and they’ll assume that their API is tested because they’ve tested their front-end UI. The problem with that is that oftentimes our front-end application, whether it’s web interface or mobile or whatnot, they’re doing front-end validation, client-side validations. So, erroneous requests will never make it to the back-end server because our client is catching it in time. So you’re testing the client that it’s validating the input before it goes to the server, but you’re not really getting a chance to test the server. So just relying strictly on UI-based testing to exercise your API is just not sufficient enough. It’s a good start from the integration from front-end to back-end. Making sure everything’s working. But it’s not the complete story.

So the other things that I recommend is, one is looking at contract testing. This can be automated. There’s a lot of tools out there. Now they’re starting to merge the two. If you capture your API contract or you’ve got a description and say OpenAPI Spec, what used to be called Swagger, as you mentioned earlier. If we use that, there are some tools that will read that description, which is machine readable, and will turn it into a starting point for a suite of automated tests. There’re some commercial tools like that. There’s a few open source libraries and others that can help make sure that the API is doing what it says it will do. That your code matches what the spec says that’s captured in OpenAPI Spec. The other thing you could do is integration or acceptance testing. I prefer to really just focus on acceptance testing, which takes the job stories and the activity steps from our Align phase of the ADDR process, and turns those into executable code. So those that are familiar with Behavior Driven Design, BDD, things like Cucumber, frameworks like that are probably familiar with the when, then and so on, kind of those types of formats. You can likewise take the work that you did upfront during the Design and the Align phase, and turn that into acceptance tests that can verify that all your APIs, when combined together can produce the outcomes that you initially defined in the job stories during the Align phase.

So you’re taking all the work that you did up front, and that’s going to, in turn, drive your testing to make sure that your API delivers on what you initially identified in the Align phase is what the API needs to do. So doing acceptance testing around our API in that way is really valuable. Some teams will do it with using tools like you mentioned, or like Postman and others that have collections that are built up that exercise each step and maybe script the transition from one call to the next. Pulling data along the way. So the output of one call turns into the input of the next call. So they can make those orchestrated calls and make sure that things end up in the final outcome state that we want, where they might write it by hand or some combination to do that. So those are just kind of the high level ideas I have there. But there’s more, that’s kind of detailed out in the book. But it’s really important to make sure you have a solid testing strategy. And if you do this design first approach, then you have all the artifacts you need to help inform you of what kind of tests you need to write. You just need to write them. So it really makes a lot of sense to be able to pull all that together.

[00:47:27] Henry Suryawirawan: Thanks for sharing some anti-patterns like, you know, testing from just the UI. So I think here also, it’s applicable to apply the test pyramid principle, right? Where you don’t just have all the UI based testing, but you have a lesser integration or acceptance test part of the UI part. Thanks for sharing that.

[00:47:44] 3 Tech Lead Wisdom

[00:47:44] Henry Suryawirawan: So James, it’s been really great to talk about all these APIs. I learn a lot. All these principles and the tools and also the concept that you mentioned about like job stories, ADDR. Unfortunately, we have to end this conversation. But before I end this conversation, normally I would ask all my guests to share the three technical leadership wisdom. So this is mainly for you to actually advise other people based on your experience or your knowledge. So what will be your three technical leadership wisdom?

[00:48:10] James Higginbotham: Yeah. I think the first one, and it’s the one that I try to really incorporate into my workshops, is be curious, not only of new technology, but of our software development history. I think we tend to always strive to see the new and interesting, and the shiny new things. But if we go backwards a little bit sometimes, and look at what people have done in the past, and learn from those that came before us, there are a lot of lessons we can learn that we can then apply as we move forward with software development. There’s a lot of wisdom out there and sometimes we just ignore it because we feel like it’s maybe too old or are not relevant today, but a lot of it still is relevant. Even though maybe the technologies have changed. We’re doing more distributed computing than we used to. There’s a lot of history out there that I think people should be students of, and so I try to incorporate that to my workshops a little bit as I go, just so that people can gain that insight if they haven’t been exposed to it previously.

The second thing is to always evaluate new technologies in light of what we’ve learned from that history. Oftentimes, we’ll see new things come up. Development team puts something together, a new open source project put something together and they think they’re the first one to have ever done it. In reality, they haven’t. In fact, one story is that I’ve had a client that ended up reimplementing a message broker over HTTP. So they effectively got pretty close to AWS’s SQS, and they didn’t even know it. They didn’t know those technologies existed. You know, it’s no fault of theirs. All of them on that team had never really had a need for messages oriented middleware, or traditional message brokers, think RabbitMQ, things like that. They just never encountered it. And so they ended up reimplementing the same thing themselves, but it was a little less reliable. So we re-evaluated that, and consider how we could learn from technology that has been around and what will be the best fit for them.

The third thing I would say is pass on what you learn to the next generation. I think there’s been several studies that have indicated that a half-life of a developer in our industry tends to be around five years. So that means that we’re not really always doing a great job of helping the newer generation become more familiar, not only with our history and not only how to evaluate new technologies. But just the wisdom that we’ve gained in learning how to code. So the software development can be a very solitary role at times. We can put our headphones on and get into that zone and really have a good time. Writing code that might be our passion and I enjoy that as well. But it’s important to step out and help others, and listen and learn from others as well. So, passing on what you learned and then learning from other people, that’s the third one.

[00:50:35] Henry Suryawirawan: Thanks also for spending your time here to pass on what you know to all those software developers who are still learning or maybe developing web APIs. So thanks James again. For people who want to follow you online, or get to know more about your work, where can they find you?

[00:50:49] James Higginbotham: My company website. You can reach me and find articles that I’ve written and other insights. It’s at launchany.com. That’s L A U N C H A N Y.com. That’s my consultancy. I also have a newsletter called API Developer Weekly. So you can go to apideveloperweekly.com. It’s a weekly hand curated newsletter, where I focus on a lot of API related articles, and non API articles whenever I find something interesting. So I just send it out once a week. It’ll keep you up to date on what’s happening and interesting developments, case studies, other things that get read by other people. I share that out every week.

[00:51:24] Henry Suryawirawan: So I’ll make sure to put that in the show notes. So thanks again, James. I wish you good luck with all the consultancy and helping people to build a proper API.

[00:51:32] James Higginbotham: Thanks Henry. Appreciate it.

– End –