#214 - Beyond CI/CD: Continuous Deployment Explained - Valentina Servile

[00:01:35] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I have with me a lead software developer from ThoughtWorks and also the author of a book titled Continuous Deployment. So as you can tell, we are going to discuss about continuous deployment. How does it differ from, you know, what we know, CI/CD, continuous integration, continuous delivery. And yeah, happy to have you Valentina in the show.

Valentina Servile: Hi, Henry. Happy to be here.

[00:02:00] Career Turning Point

Henry Suryawirawan: Right. Valentina, maybe before we go dive deep into continuous deployment and all that, maybe let’s talk a little bit more about yourself. Maybe if you can share any career turning points that you think we can learn from you.

Valentina Servile: So, well, you already gave a bit of an introduction. As you said, my name is Valentina Servile and I am currently a tech lead at ThoughtWorks. I’m Italian as you can tell from my accent, but I’m actually working in ThoughtWorks Spain. But I did start my career in Italy. So basically, I began by joining a very small Agile consultancy. That was my very first job and I had a very amazing training. So all of the juniors that joined this consultancy basically were told to study for three months. As soon as I began on my first day, I got told to read the Agile Manifesto, which is something that I still really believe in, you know, all of the Agile principles. And eventually led to me writing a book about one of those. So, yeah, maybe this is a very good reminder to know. Pay attention to how you’re training your juniors, because it really does pay off, I think, if I can use my examples.

Three years later, I joined ThoughtWorks in Spain, and I started working with much bigger companies. I started working with more enterprise type clients and, you know, being a consultant in both occasions, I was able to jump from client to client to client and see a wide variety of ways of working of different team compositions of different practices. And I think that gave me a really good perspective on, uh, all of the technical practices that teams can adopt to be successful or not to be successful. I also, in between that, managed to go on a long term assignment in Asia. So I was stationed in Bangkok for about a couple of years. And so it was really nice as well to see perspectives of how organizations work on the other side of the world, in my case. But yeah, now I am back in Spain. I was back in Spain for a while. Now I wrote this book. And let’s see what’s next.

Henry Suryawirawan: Wow, thank you for sharing your story. I think one thing that you mentioned earlier, right? So train your juniors.

[00:06:05] Tips for Juniors Starting Their Careers

Henry Suryawirawan: Well, I think maybe not all juniors are fortunate enough to have this kind of a good training, good principles being taught since the very early of their career. So maybe for those who are maybe not so fortunate like you or maybe some of us. So what advice would you give to them such that even though they don’t get the good training from the company or from the seniors they work with. How can they still learn a good principles at the start of their career?

Valentina Servile: I’m going to say, well, first of all, try to look at the literature because that has been really, really helpful for me, you know, in terms of good practices, in terms of how to be a better programmer. The literature has so much insight even if you cannot get it from your seniors or from your mentors, you can always fall back to that. Just don’t be afraid to ask questions and be annoying, you know. Try to be the one with the keyboard when you’re pair programming, for example. Yes, it will be a bit slower. Yes, you will have more questions than someone else would and hit more blockers than someone else would. But make the best of your mentors being there and don’t be afraid to annoy them because that’s the only way that you get to learn, you know, being hands on.

Henry Suryawirawan: I like your term, be the keyboard, right? So sometimes we may not be comfortable asking questions to senior and become like a, I don’t know. Some people think they might be a hindrance to the discussion and all that. But I think, yeah, beginners mind, I think that’s always a very good skills to have in any kind of team.

Valentina Servile: I was gonna say and it definitely helps the team as well. You know, you might think that you’re annoying everyone by asking questions. But a lot of times it actually uncovers something more to the discussion, even what might be a really obvious question. So yeah, definitely don’t feel like an annoyance. What you’re bringing is really, really valuable, which is a fresh perspective.

Henry Suryawirawan: Right. I think that’s really great tips, right? So for all the beginners, even maybe for seniors, right? Sometimes don’t be afraid to ask like so called stupid questions.

[00:08:00] Continuous Deployment Book

Henry Suryawirawan: So let’s go dive into your book, right? Continuous deployment for me, it’s almost like natural one day somebody will write about this continuous deployment. Even though, you know, like CI/CD has been, you know, around for, I don’t know, more than a decade, I guess. So maybe in the first place, tell us your story, the background, what inspired you to actually write this book?

Valentina Servile: Yeah. And actually I brought the book. It’s right here, uh, Continuous Deployment. You can get a physical copy on Amazon, of course. So what inspired me to write it? Basically, as you said, it sounded like a really obvious gap in the literature that somehow was missing. I don’t know how nobody had thought about writing this book yet, but it’s definitely a gap that I felt when I was working. Because being a consultant, I was really lucky to be part of some teams at some clients that were already doing continuous deployment. They were doing it for many, many years.

But sometimes we were onboarding more junior people or consultants that didn’t have this experience. And even when they were really familiar with other great practices like continuous delivery, continuous integration, DevOps, there was still a bit of a learning curve. So I found myself basically in a situation where I wanted to communicate what is the difference between stopping your commits at staging and every commit going to prod. And it turns out there was quite a lot to communicate about it. And I had basically no literature that I could just send them or articles that I could just send them to, you know, just read up about it and be ready the next day. And so yeah, based on missing this kind of content, I decided to create it.

So I started by writing some blog posts and doing some talks about the subject. And yeah, then it got picked up by O’Reilly and here it is now. So hopefully this will help other people who have been in my situations, for example, in teams where you’re already practicing continuous deployment and you want to onboard someone else or if you’re one of the people joining such a team. So this is a pretty condensed version, even though it’s 400 pages long of everything that you might ever need to know about it.

Henry Suryawirawan: Yeah. And you also got Dave Farley to write the foreword of the book, right. One of the co-authors of the, you know, I would say seminal book, Continuous Delivery, right?

[00:10:16] Definitions of CI, CD, Continuous Deployment

Henry Suryawirawan: So I think maybe let’s start from there, right? The definition. It’s always very confusing when we talk about, you know, CI/CD, in the industry, right? Some people even already associate CD with continuous deployment. I guess maybe a little bit of back to the fundamentals, if you can elaborate what are continuous integration, continuous delivery and continuous deployment.

Valentina Servile: Thank you. So this is a topic where the terminology has undergone a lot of semantic diffusion as Martin Fowler calls it. And a lot of people in the industry refer to CI and/or CD as their pipeline effectively, forgetting that in reality, all of these things started as practices. So the pipeline is just an artifact that helps you realize the practice. But there is quite a lot more to it.

So starting with some definitions. The first practice that came along was continuous integration as part of extreme programming. So the practice behind that, outside of the pipeline tool, was basically the act of integrating your changes into a mainline shared branch really, really frequently. This had the purpose of always having all of the changes bundled into a buildable artifact that could be validated by tests, for example, and so that the team could always show their progress. So it came with practices such as nobody pushes on a broken build, we have to integrate changes frequently, not hoard changes in really long lived feature branches, and so on and so forth.

Then the pipeline, which is how the implementation looks like or can look like of continuous integration, takes care of taking this mainline branch, building an artifact from it and running some automated tests. And that was a really big change at the time because there wasn’t a particular way of working that existed so much before. And suddenly it was really, really easy to show progress, and say, look, this artifact represents what we have been working on so far. And it is buildable, it is testable, and it can be deployed because it contains every change that has happened, say, in the past day or in the past week. That made the integration much more frequent and introduced the concept of the pipeline.

But then continuous delivery came along and it brought even more to the conversation. So continuous delivery added on top of continuous integration the fact that not only your changes should be frequently integrated, frequently tested, frequently built, but also frequently deployed. And every single change that goes through the automated pipeline should be verified to such a greater confidence level that should be considered deployable to production. So basically this was a really big mindset shift because then it came with other practices, say, from DevOps, such as infrastructure as code, and automating the deployments themselves, which didn’t have to be manual anymore. But yeah, it was also a practice before it was the automated pipeline tool.

Then a further next step on top of continuous delivery again to me and to many other people has been continuous deployment where not only every change that goes through the pipeline should be considered a deployment to production candidate, but it should be actually deployed to production. So basically, this to me was a combination of this concept of the automated deployment pipeline, where every single step between development and deployment to production has been automated. In the traditional implementation of continuous delivery, for example, you might see that many, many teams deploy automatically with their pipeline to some pre-production environments, say, dev or staging. But the deployment to production, while it can be done on demand, is still a manual action. So it’s the push of a button to be done only when the team feels confident enough.

Let’s say continuous deployment is pushing that limit away and removing that final manual gate in the value stream of building software and deploying it to prod. And so, yeah, it is definitely a culmination of a journey that has maybe taken, I’d say, 20 years since extreme programming and continuous integration came along, and it’s bringing automation all the way to the end of the value stream.

Henry Suryawirawan: I like that you brought up the historical aspect of how all this culmination happened, right? So from things like extreme programming, you know, DevOps, Agile, and things like that. And I like the specific thing that you mentioned, right? It’s not all about pipeline, right, because people associate maybe CI/CD with pipeline concept. Even tools like Jenkins, maybe now GitHub Actions and things like that. But it’s about the practices, right? The practices that actually lead us to use some of the tools that could help us actually adopt those practices.

Maybe slightly a little bit more about definition as well. People associate CI/CD, right, they think they are kind of like a different thing. And now what is the term that you’re going to use for continuous deployment? Is it also the same CD or is there any other term?

Valentina Servile: Actually, that’s not really set in stone yet. I’ve been debating it with my colleagues. Some of them abbreviate it to CDep just to distinguish it from continuous delivery. So if you find it helpful, I encourage you to continue on that. But yeah, otherwise you can just spell it out, continuous deployment, just not too confusing.

[00:15:42] If It Hurts, Do It More Often

Henry Suryawirawan: So I think one aspect that is very, very important that I would like you to also try to explain, right, this thing in the extreme programming, right? If it hurts do it more often, right? So I think this is maybe one of the key reasons why continuous delivery was kind of like invented in the first place. Maybe tell us why this is important for us, especially for those teams who are still kind of like fearful whenever they make code changes and they want to deploy it even to production.

Valentina Servile: So I love that principle from extreme programming. To me that is the essence of extreme programming and most of agile practices themselves, to be honest, is software has many more parts to it than just writing it. There’s integration, there’s testing, there’s deployments. And all of those other parts before XP used to be pretty painful. So, you know, in an old fashioned waterfall process, you will maybe have your giant development phase, then a big integration phase, then a big QA phase and so on and so forth. Now that used to be pretty painful, because it was done infrequently, right?

So the principle of extreme programming basically says, if something is painful, such as these phases of writing software, you should do them more often, because doing them more often encourages their automation. which is a great thing in and on itself. Automation allows you to, first of all, document and also make a process which can be replicated. It is not relying on someone’s knowledge on doing things by hand. But also more crucially, if you do things more often, like integrating software, testing software, there is less software to integrate and test and deploy each time. So when your batches are smaller, and here maybe I’ll talk later about lean manufacturing, suddenly all of those phases become much less painful. And that is why this advice seems counterintuitive, but actually it allows to feel less pain in performing all of those phases.

That said I believe that continuous deployment is the perfect example of applying this principle and putting it into practice because to me, there is absolutely nothing I can think of that is more painful than deploying to production for teams, right?

That is so much more painful than resolving some Git conflicts, so during integration or spending a lot of time on testing. The deployment to production is when the website can go down. Your app might stop working. You could scrap your database. So because it is much more scary and much more painful, that’s why I recommend to do it as often as you can. And in this case, automatically at every commit.

Henry Suryawirawan: Yeah. So I think in the past, traditionally, I also used to experience those big bang release, right? Where you kind of like have a certain window, maybe quarterly, whatever that is. And you kind of like build a ceremony around production deployment, right? Kind of like fearful, whenever the new change going to be used by the real users and whether there’s going to be a big impact or bug happening in the production. So I think the spirit here, if it hurts, right, do it more often so that your production deployment becomes like a more boring kind of event.

Valentina Servile: I was gonna say, whole ceremony about deploying to production. The whole do not deploy on Fridays. That’s not something that you should be prideful about. That’s actually a smell that your process might not really be so great. If a deployment is that risky, it means maybe there’s something going on that you need to address. Perhaps do it a bit more frequently and split it in smaller, smaller deployments.

[00:19:18] Why Remove The Final Manual Gate to Production

Henry Suryawirawan: Yeah. Speaking about the difference between continuous delivery and continuous deployment. I mean, one big aspect that I see is the big differentiator is what you call the still kind of like the manual deployment to production, right? So in the past we used to have like this manual approval stage where, you know, somebody will decide when we actually deploy to production. So tell us why it is very, very important to actually remove this last critical piece. Because for some stakeholders, maybe they think having that is kind of like a guardrails to ensure things are still going to be safe. It’s kind of like verified before things are deployed to production

Valentina Servile: So to explain why it is important to remove these which might seem like a small piece and a very important guardrail. I’m going to connect to the practice of lean manufacturing, right? To me, really the most foundational practice of Lean is removing inventory from the value stream of software. So what we see as the founding principle of Lean is that instead of having big batches of inventory going in between workstations, we should try to reduce the flow of inventory that is going through the system to basically be only one piece at a time. This allows to do some crucial things, which is to, for example, identify defects immediately as they flow through the value stream, rather than waiting for an entire batch of things to be processed and going to the next workstation. But also, even more crucially, it allows to drastically reduce the cycle time. So the time it takes for any piece of work to flow through the entire stream and be in the hands of customers.

Now how does this translate to software? This is a principle that was very successful when applied to manufacturing of physical goods. But of course it’s very hard to define inventory in software. So what I think of inventory in software is pretty much every intellectual artifact that is included in software manufacturing. So this for example could be user stories. It could be architecture diagrams, or it could be code. So I’m gonna focus mostly about code here. How to define a batch of software, for me, it needs to depend on how to identify a unit of software. Defining a batch of software basically has to depend on defining a unit of software.

So what is really a unit of software? Like, when do we know that we have a big batch of inventory? A unit cannot be really defined as a single instruction in code. I don’t feel like because that’s not self consistent. Instead, I define a unit of code as a commit because that is the smallest unit that we can have flow through that value stream, in this case, the value stream could be represented by our pipeline. So a commit is the smallest possible unit by which we can review software, test software and eventually deploy software. So to me, a batch becomes when commits accumulated during this value stream, right? So when commits accumulate in your pipeline, that is when you have some inventory that is stuck at a specific phase of your value stream of code. And that is where defects can accumulate. That is where your cycle time is slowing down, and that is where all of the anti patterns of not doing Lean start to emerge.

So bringing this back to continuous deployment, you can think of continuous integration and continuous delivery as the gradual transition from an older batch and queue processes, such as waterfall, for example, where there is a lot of inventory moving really, really slowly in between workstation. Continuous integration and continuous delivery represented a shift from that. So with continuous integration, you’re starting to have what looks like a one piece flow from the development phase, at least to the integration phase, because it says integrate frequently. So how frequently can you integrate at every commit, of course, and how frequently can you test at every commit? So you have commits suddenly flowing one by one through at least some of the earlier phases of software manufacturing. And so your inventory there is checked with the maximum granularity that it can be checked. Of course, later than those phases, so say, the deployment phase wasn’t yet automated in continuous integration. So that’s where commits or inventory could accumulate right before a deployment. And basically it would go through the rest of the process again in batches.

Now with continuous delivery, the deployment phases following that have been automated. So again, this one piece flow of inventory had been extended. But still with most implementations of continuous integration, commits still accumulate in pre-production or in staging. So that’s where, basically again, your defects start showing up and your commits start accumulating right before production. Now, this makes deployments to productions riskier because, of course, you have a bigger batch. And so any defect that has accumulated in that batch is going to show up upon deployment. So I think, if you want to realize this one piece flow all the way end to end, you shouldn’t have any manual points in the pipeline where these commits can accumulate and become stale.

Henry Suryawirawan: Wow. I think it’s a very nice explanation why we should work towards this manner, right? This practice, right. So one piece flow. I think that’s a very important, thing in from Lean, principles, right?

[00:24:56] Common Challenges in Adopting Continuous Deployment

Henry Suryawirawan: So I think also maybe for some people who are still not yet in this kind of a working mode, right? What do you see some challenges maybe from your consulting? Maybe from, you know, having the chance working with enterprise, mostly, I believe. They will have a lot of mindset shifts that need to happen. What will be some common challenges? Because deploying every commit straight into production is not something that all the team can aspire to do, right? Some might have regulations, some might have compliance checks. Some might have any gatekeeping inside the software delivery. So what do you think are some common challenges that people need to really start to move away?

Valentina Servile: Yeah, and what you just said is exactly why I have at least two chapters of disclaimers around when can you really do continuous deployment, when can you not. Surprisingly, there’s a lot of situations where you think you might not be able to do it, such as regulated industries. But in reality, you’re able to. It depends really on the framework and regulation.

For example, in my case studies, I have a neo bank called N26, which is subject to basically all core banking regulations, but they were able to still use a continuous deployment workflow, because they have an automated pipeline that basically satisfies all of the auditability requirements of the process. And as Dave Farley says in his excellent article, Continuous Compliance, an automated pipeline is an absolute gold mine for auditors. Basically, it has so much information on which changes were applied to production, when, by who, for what reason, because it will be related to a certain ticket, what tests were they performing on them, how did these tests prove that they were successful, and so on and so forth. So that already really helps. And it’s just a matter of figuring out what parts of your process can really apply to satisfy this regulation.

At the same time, it’s very common in regulated environments to have a four eyes rules where nobody can really put a commit in production alone. That is something that also this bank was able to satisfy because they found a way by doing a combination of pair programming and micro branches with like immediate pull requests approved by a pair. That was their way to prove that a commit was seen by at least two people. The person that authored the commits and the person that merged. It might sound counterintuitive, but there are ways to still use modern practices in regulated industries, not because these practices are less safe, but because they are more safe than manual approvals and waterfall. And we just need to find a way to articulate that and really explain that.

Other situations where it might be difficult to perform continuous deployment are usually dictated more by technical challenges rather than regulatory ones in my experience. So for example, something that is really common that happens to teams that maybe want to move from continuous delivery to continuous deployment is, uh, when you start doing deployments really, really frequently, you might find out that your system is more sensitive to deployments than you might think.

And so this might affect things like your reliance on caches. Suddenly when you deploy many different versions all in the same day, your caches, if you have in-memory ones, are going to be mostly cold, most of the day. So that’s when you need to find maybe architectural workaround, such as moving your caches externally and so on. Also, this applies to any kind of in-memory state that you might have in your application. So if your user flow depends on certain state being stored in memory for whatever reason, that is something that frequent deployments are going to be disrupting really, really frequently. So you need to also fix those kinds of scenarios before you can really move on to this workflow.

Another really common scenario where people have trouble and maybe, this one is also easily fixable is when your production environment that you want to continuously deploy to doesn’t belong to you. So often we think of production as a server that is in our hands, and it is in our capability to deploy to.

But what about, for example, mobile apps, or IoT devices, or on premises infrastructure that belongs to someone else, right? In that case, it’s a bit more of a gray area. What does it even mean to be done with a deployment to production? Are you done with your deployment when the first user device has been updated or when the last one has been updated? So there that’s when, uh, probably continuous deployment is not so easy to implement. But again, there are workarounds that you can use. So for example, uh, you can use PWAs to move control back to the server. Web views. But yeah, that’s, uh, when you may struggle a little bit.

Henry Suryawirawan: Yeah, thanks for the overview of some of the challenges, which I assume some teams would have, right, whenever they want to try to implement this continuous deployment.

[00:30:02] Minimum Practices for Continuous Deployment

Henry Suryawirawan: But having said that, there’s also another team which probably claim that they do continuous deployment, which I want you to kind of like explain. Because they may not have so many processes, they just have a commit. And they maybe do something in the build pipeline, right? Maybe creating the artifact and just deploy to production. I’m sure this is not something that you are advocating, right? Continuous deployment doesn’t mean that you just deploy straight away, but there are certain processes in place. So tell us what is the minimum thing that you should have in a continuous deployment?

Valentina Servile: Yeah. And thank you so much for asking that question, because I don’t want anybody to take as a takeaway from this interview that whatever the state of your automation now, you can just get rid of any manual approval and go straight to prod and nothing will happen. So basically, there are a certain minimum amount of practices that you should definitely make sure that you have. So again, a pipeline that is fully automated. But that pipeline needs to contain tests and a lot of them. So that’s what I go back to recommending, you know, the typical testing pyramid, which means you should have a pretty, pretty good unit test coverage, a pretty good coverage of automated tests both before and after deployments, if that is possible for you.

There are practices that can maximize this coverage. And the ones that I use most often are the ones that basically allow you to write your test first. So TDD is my prime example. And outside in TDD. If you use this, you know that your coverage is growing together with your code in the same commit, which is then going through the pipeline. So those can make you pretty safe. Then alongside tests, which are great for functional changes, I always recommend some forms of static code analysis to be in your pipeline, such as security scanning, performance scanning. If you use a tool like SonarQube, that is going to be really effective at catching most regressions that maybe might be missed by your tests. And that’s really important, right? Because if you’re publishing every commit, then every single commit might affect the performance of production or open a security vulnerability. So static scanning can really, really catch all of those regressions at that really small granularity.

Then, of course, another question that I get really often about what are the prerequisites is how do you hide your work in progress if you’re just publishing every commit to production? Because you say you shouldn’t have long feature branches where you hide your changes, but also everything on master needs to go to prod immediately. So we need to also be comfortable with our team to use patterns like feature flags and the typical expand and contract pattern or parallel change or branch by abstraction, whatever you call it. All of those patterns can help you hide any new change that you don’t want your users to see, even though that change is in production. So I highly recommend to be familiar with those.

And then finally, of course, you need to have zero downtime deployments as a minimum practice. I think most teams do these days, but it’s always best to check. So you can achieve those with rolling updates, you can achieve it with blue green deployments or red black. You can also perform canary deployments with tools like Spinnaker, where basically the tool is going to be connected to your observability tool and observe, for example, the error rate of your application and roll back immediately if that error rate becomes worse.

So observability and alerting is maybe the final absolute prerequisite that I recommend before turning on, you know, the fully automated pipeline and moving the last gate to prod. Because as deployments get more frequent, there is also less oversight on them. As you said, they become just a boring event that happens multiple times a day. So we need to be proactively notified in the very rare event that something does go wrong. And that’s where I recommend having a really, really sturdy dashboard, keeping track of all your metrics. And most of all, take care of your alerting based on your metrics, because you definitely don’t want your team to be used to seeing a lot of alerts and not having to react to them. So keeping those really tidy and only keeping alerts that are useful and require any sort of action that’s gonna really help.

Henry Suryawirawan: Thanks for the clarification. I mean, just to re-emphasize one more time, continuous deployment doesn’t mean that any kind of commit you just deploy to production straight away, right? So there are certain practices, certain minimum that you need to achieve, right? And Valentina just mentioned, you know, several of them, right, which I believe are like some core practices that every engineering team should aspire to do.

I think one aspect of continuous delivery that is also very important is that everything that you prepare to go to production, right, must be verifiable and you have high degree of confidence that it will work in production as well. So it’s not something that you kind of like YOLO, you know. You just wait for the users to report the bugs to you. And I think one other aspect is about backward compatibility, right? So whenever you make so many changes, right? I’m sure some users, maybe some servers are not taking the changes straight away, right? So you need to make sure that some backward compatibility must be in place.

[00:35:17] Hiding Work-in-Progress

Henry Suryawirawan: Which brings me to this thing that you mentioned, right? Hiding work in progress. Definitely, if every commit must go to production, then we cannot finish a new feature just within one commit, right? So you mentioned about feature flag or maybe expand and contract. Tell us more a little bit about this practice because I’m sure some people might still think, oh, I cannot work this way simply because I have some work that I can’t finish. So tell us how we can use those techniques to do this.

Valentina Servile: Yeah. Thank you for mentioning those. They are the ones that I rely on most heavily throughout the book as well. So I think it’s definitely worth to take the time and understand them. So let’s start with the feature flags. So I recommend using feature flags whenever there are visible changes to the code base and the application that if deployed to production will alter what the user is experiencing and the observable behavior of the system. Most people are used to hiding such changes in version control branches. And then the act of publishing them will be to merge the version control branch into main and finally let main go to production.

Using a feature flag instead still uses a kind of branch, but it is not a branch that belongs in the version control system. It is a branch that belongs to the tree of classes and functions within your application. So I call it an execution branch for that reason. And that will do exactly the same job of a version control branch, except it’s going to still allow you to integrate your code frequently and deploy your code to production frequently. So during development, you will basically put a feature flag as the very first step of beginning any new feature. And that feature flag will be off until that feature is released to the users in production. I think this is really powerful because it also allows you to express the conditional logic of when you want your feature to be seen or not seen, for example, for a release in a way that is much more expressive than just the textual control that version control gives you. So I really rely on feature flags a lot for that reason.

Finally, for expand and contract, that is a pattern that I tend to use when I’m making not so visible changes, but changes that I still, I’m not feeling super confident in or that I want to split into smaller and smaller increments. An example of this could be refactoring a really big subsystem of your codebase or, for example, an entire endpoint of an API.

That’s when I would basically duplicate the flow completely, making sure that the interface of the old flow and the interface of the new flow is respected equally. That allows you to basically commit at any point, because you can migrate the clients of the old flow to the new flow one by one and still completely respect the functionality of the system, which your regression tests should be able to catch if you broke that. So yeah, you can read more about this in the book. I have a lot of concrete code examples.

Henry Suryawirawan: Yeah, I highly recommend reading this part, because I think this is one key practice that is very important in continuous delivery and continuous deployment, right? So for those of you who also have not heard about expand and contract before, some people call it branch by abstraction. Maybe if you have like OOP, you have interface implementation, different abstract class and different implementation. This is also one aspect that is kind of like similar in terms of implementation.

[00:38:46] The Difference Between Deployment vs Release

Henry Suryawirawan: And I think this is also a good time to actually explain the difference between what we call deployment versus release, right? Because with things like feature flag, it allows us to kind of like control the time to release a particular feature. So tell us about this difference.

Valentina Servile: Yeah, and this is the reason why I love feature flags, right? So with continuous delivery, it was still true that you could use feature flags and have a separation of deployment and release. But with continuous deployment, it becomes mandatory. And the use of feature flags becomes mandatory because your pipeline is not going to stop anything visible from going to prod if the tests are green.

Because those practices are mandatory, the distinction between deployment and release becomes much more well marked and much more important for teams. So a deployment, in itself during continuous deployment is any rollout of new conversions into production. But a deployment in and of itself should never alter the visible behavior of the system.

At the same time, a release usually becomes the act of turning on a feature flag, which again, if you have a good feature flag framework should never require a deployment because you can do so at runtime. So there is a very, very clear distinction in the timing and how these two things happen. A deployment becomes just an engineering concern, and release becomes a product concern. It’s like, when is the right time to turn on this feature? Do we want the entire user base to see it or do we want to run a 50:50 test first? Or do we want to do a gradual ramp up? How does product feel like this would be best released?

And this also brings us to a really great situation where a deployment being purely an engineering concern can be done without the input of product. So we should never be in the situation, say, where I, as a developer, get told, please do not deploy any new version, because we are in the peak season, for example, for our business and the product wants to wait. At the same time, I should never be in the situation where, as a developer, I have to tell product that feature cannot be released because there is a pending deployment, because all the code should have been in production in the first place.

So, yeah, definitely, these two things kind of take a life of their own with the use of feature flags. And that is why I consider them such a strong, strong prerequisite for any modern pipeline, be it continuous deployment or also continuous delivery.

Henry Suryawirawan: Yeah, I think such a good distinction, right? Deployment should be the technical or engineering concern, while release should be the product or stakeholders decision, right? If the team can separate these kind of two important activities, I think that’s a really great shape. And I think a feature flag, so many tools these days, I’m sure having a platform that can allow you to do this behavior is something that is not impossible anymore.

[00:41:40] Slicing the Work

Henry Suryawirawan: Another important thing that I think worth to call out here is about slicing the work, right? Because if let’s say I want to build a new feature, sometimes a feature is not like very simple, like change a color of something or create a button or something like that, but it’s more like a maybe like workflow or something like that. How do you actually advise us to slice the work, right? Because making sure that the small batch committed, up to production, is something that maybe some of us are not used to.

Valentina Servile: That’s a great question. So there’s always been that eternal battle between slicing work by code base and slicing work more vertically by feature. I’m going to say, with this type of workflow, continuous deployment is even more important than it was before to start thinking of slicing work vertically by feature instead of by code base. That is for the very simple reason that, as I mentioned, we have practices like feature flags and especially expand and contract that require multiple small changes to production on different systems a lot of the time to guarantee backwards compatibility until the feature is completed.

So for example, in order to add some new functionality to my system, I might need to alter my front end and back end codebase and my DB, right? Now because I’m doing continuous deployment, I want to maybe build it slowly and deploy multiple commits one at a time into production to guarantee, for example, backwards compatibility of my back end, say with my database schema. Maybe I need to create a new table, migrate something in my back end to go to this new table instead of the old one. That’s a new deployment on the back end. And then maybe I need to redeploy my database to clean up the old table, maybe synchronize some data.

So if I slice this tasks horizontally, I have to create so many tasks just to jump around on which change do I need to make now on this system and on this system, and they’re all gonna be blocking each other. So it’s not a really efficient way to work with these ways of working. Instead, if I slice vertically, I can split my task, my user story into multiple commits, but I have the freedom of relating them all to that ticket or that user story and to really plan out within that story, what commits do I really need to do in which system and in which order to achieve that functionality at the very end?

So, yeah, it becomes much, much more important to slice vertically, and it also allows you really to leverage the granularity of continuous deployment as a practice, because the fact that everything is in production all the time, it can make emerge some situations where actually you’re ready to release a feature in a state where you previously thought you wouldn’t. So say after maybe only two or three user stories instead of after all 10 user stories, because perhaps product wants to run an A/B test a bit earlier. And so not only becomes important for technical reasons, but also it’s how you really leverage the speed of continuous deployment.

Henry Suryawirawan: Yeah, so I think vertical slicing is the key here, right? It also goes back to how you slice your story, right? So a feature should be sliced into different shape of stories. And if you can shape it more vertically that brings value like any kind of value to the users or maybe some other third parties, right? I think that will be the key.

[00:45:10] Coordinating Changes Between Systems & Third Parties

Henry Suryawirawan: Talking about third parties, I know these days people are working in a kind of like microservice or service-oriented architecture. Continuous deployment also has this challenge of having to coordinate your changes with other third parties, APIs, whatever that is, right? So tell us, how can we do this better? Because obviously you can’t control how the other team is ready or taking changes from your new version, right? So how would you advise us to do this?

Valentina Servile: it depends. There’s third parties and third parties, right? If you’re working in a team that takes care of micro service within a larger ecosystem, another team within the same organization will be a third party to you. But also it could be that you have to communicate with a vendor system, which is in a completely separate organization, and they have a completely different change management process. So I’m going to say there is much more flexibility when you’re dealing with other teams as third parties, because you can coordinate much more frequently on what new version of what is going live at which point. And you might even be able to pair with someone from another team to always guarantee synchronization between the systems changing.

On the other hand, if you’re dealing with third party vendors, that’s when you need some stronger practices that are much more formal to manage the contracts in between systems. And that’s where I usually fall back to things like API versioning, which is something that I wouldn’t do necessarily. If I have an API just for internal usage, I can very easily coordinate with other teams and say, hey, please change your usage like this. And we’ll tidy up with that. They need to maintain a thousand different versions of some API that is going to be a lot of maintenance. But with an extended vendor where I have no control over what they’re doing, that’s when I would pay the price of versioning an API, for example. Same thing with contract testing. I’m going to have much stronger contract testing requirements, if I am consuming or producing an API for a third party vendor than for a team within my same organization. So that’s how I would handle it.

And of course, within the organization, that’s where practices like expanding contract and feature flags get even more useful than within just a singular system, because they not only allow to hide the work in progress within one system, but also across several systems. So they can also be used to guarantee backwards compatibility of systems that are independently deployed and where therefore you cannot bump everything to the next version exactly at the same time. So I encourage teams to look into these practices as well for working with microservices and testing integration with other microservices.

[00:47:58] The Importance of Backward Compatibility

Henry Suryawirawan: Yeah, and I would like to again emphasize backward compatibility, always ensure that you support a few versions behind, right, simply because you can’t control the other teams, whether they are taking your changes straight away, or they will take some time before they actually apply those changes, right? So I think you mentioned also contract testing and things like that. Definitely is a good practice as well.

Valentina Servile: I just wanted to add as well that this is not just something that needs to be taken care of when you have a microservice environment. The concept of backwards compatibility applies to any system which is distributed, even if at a glance it looks like a monolithic system. In my experience, any system that is interesting rarely is really monolithic, right? For example, even if you have just one simple website with one database, you already have three distributed components that you need to keep backwards compatible at all times. You have your front end which is executed on the user browser, and it’s not going to receive changes at the same time your server does. Then you have your server and then your database is a completely separate executable and most often deployed on a different machine as well.

So the thing you really need to take into consideration is inter process communication. Whenever there is such a thing, which is almost always, you will probably want to rely on things like feature flags or expand and contract at a minimum. Continuous deployment just makes it much more evident because with that, it happens all the time. You do deployments all the time and usually microservice oriented environments. But this is not a new problem. This is a problem that we should have been solving already much before. And, you know, when deployments are infrequent, sometimes it’s easy to ignore if there’s some deployment glitches. Whereas with continuous deployment, you can’t ignore them anymore, because they just happen every single day.

Henry Suryawirawan: Yeah, and this goes back again to the principle, if it hurts, do it more often. You should not, if it hurts, then you stop doing it, or you kind of like reduce the frequency of doing it, right? So I think the more you do it, the more you figure it out how to make it safer, how to make it, you know, smoother.

[00:50:05] The Required Mindset Shift

Henry Suryawirawan: Another thing, apart from technical practices, are there cultural aspects or maybe, you know, like mindset shift that people should know about before they can actually implement this continuous deployment?

Valentina Servile: Yeah, for sure. And that’s why I wrote a book about it as well. I think a lot of it has to be a mindset shift change, especially for developers, first and foremost. Imagine that any time you do git push origin main because you’re working in a trunk based development fashion. Of course, you’re not doing PRs, right? Every time you do that, seven times a day, your users are on that life system that is going to receive that update 20 minutes later. Suddenly, if you were using continuous delivery before, it can be very paralyzing to think about what you’re putting in the pipeline at any given time. But it doesn’t have to be, right?

So there needs to be this mindset shift of putting production as a first class citizen every time you’re writing code and really think through before you pick up a new task or a new user story, what commits am I going to put in my pipeline that are going to make that happen? And which subsystem do I need to start from to not break production at any point? For example, if I have a database, backend, and front end, that is going to be probably the database system that I’m starting from, if I’m adding something new. Otherwise if I deploy the front end first, and that’s what I start developing, there’s nothing underneath to make the feature work, is all going to be broken.

Every single aspect of your work needs to be thought as a live change in a production environment rather than a plan for a change that might happen somewhere later and might be deployed by someone else at some specified point in time. So it feels much more like being an electrician working on an office building that is currently in use, right? You have to always be careful of which diversions you’re installing, how you’re going to maintain compatibility with your system to not disrupt this living beast that is production. And that also includes not only functional changes, but also security, performance.

There’s going to be a lot that you were used to thinking about maybe when your changes are in staging and you have to now anticipate all that thought during development, which to me is actually really positive. It can be overwhelming, but once you’re used to it, it means that you’ve fully shifted left all of those quality aspects, all of those cross functional requirements, for example, that maybe used to be relegated more towards the end of development.

Henry Suryawirawan: Yeah, so I like the concept that you mentioned. Always make sure that deploying to production is a first class citizen, right? So ensure that you think whenever you make change, it’s gonna be straight away go to production, right? So sometimes this requires you to carefully think how you actually implement the changes and things like that, including the shift left. I think shift left still kind of like the correct mindset for implementing this, right? As much as possible automate as well those checks so that you know, the pipeline can just automatically deploy to production once everything is verified working.

[00:53:16] AI Caution in Continuous Deployment

Henry Suryawirawan: So one aspect that is very trendy these days is about AI. Anything related to AI that can help continuous deployment practice?

Valentina Servile: I don’t know whether it can help. Maybe it’s more something to be aware of and slightly concerned about. So especially for teams that use continuous deployment today, I see that it is very common right now to just adopt Copilot. I’m one of the culprits of this, by the way, I’ve recently installed it in my IntelliJ. I just have it generate some code for me from time to time when I can’t be bothered to look up some configuration or maybe I can’t be bothered to do some test coverage. I ask it to give me a nice suite of tests for a certain function. But yeah, of course, if you’re continuously deploying, it means you have to double check, triple check, quadruple check anything that the AI is giving you in order to be fairly sure that it’s going through the pipeline.

So one heuristic that you could use is, for example, perhaps the AI could generate the production code, but maybe you should be the one writing your tests so that at least worst case scenario, your pipeline will fail. And in general, again, all the practices that I’ve recommended before as base prerequisites, such as test coverage, static code analysis, observability, those are also going to be a really good safety net. And yeah, you should really double, triple check that your pipeline is really going to catch any and all types of regressions if your team is relying on AI really heavily.

That said, it’s a great tool. It can speed you up even more, but please be careful with it. Take it with a grain of salt, whatever is spitting out on your screen because it is very easy to make mistakes if you’re just get used to not double checking it and just trusting it blindly.

Henry Suryawirawan: Yeah, I think that’s a really great advice, right? Because AI, one thing for sure, it can help us churn out more code. But more code doesn’t always mean good, right? Sometimes you have to check, especially if you don’t have robust pipeline, robust automation tests and things like that, right? So always ensure that the principle is actually the one that matters the most rather than churning out the code.

[00:55:35] 3 Tech Lead Wisdom

Henry Suryawirawan: So Valentina, it’s been a pleasure talking to you. As we reach the end of our conversation, I have one last question I’d like to ask you, which I call the three technical leadership wisdom. Just think of it like an advice you want to give to the listeners here. Maybe if you can share your version.

Valentina Servile: Sure. So I thought a bit about it. I would say, especially relevant to today’s episode, we’ve been talking about like really modern practices. I think continuous deployment. There’s nothing faster and more modern than that, that I know of in software development as like ways of working goes.

One of the main pieces of advice that I like to give is many tech leads and developers tend to get sad if they can’t use all the latest and greatest in terms of ways of work in tech. And it’s really sad because I believe it’s not necessarily a must have, right? Having everything perfect all the time. I think as long as the current practices of the organization are improving, that is a good thing in and of itself. So don’t be worried so much about the status quo. Be more worried about the trend. Is the trend an improvement or are we getting worse? And that’s also true for people who are doing continuous deployment and really nice and modern things. You know, are you getting better at it every day? Or are you slowly getting worse even though you’ve enabled it some time ago? So worry about the trend would be my first piece of advice.

The second one is sometimes as developers and tech leads, we get emotionally attached to solutions. Again, I think this is a bit of a pitfall and I believe our job is maybe more about informing. So informing of all different options and then let stakeholders determine what is their preferred one. As long as they are aware of the pros and cons and they acknowledge any risks, let them do it. Don’t take it too personally.

And finally, yeah, my main piece of advice is just if you’re a tech lead like me, just trust the people that you’re working with. Trust them to have thought about the edge cases, trust them to do their spikes, trust them that they’re working and doing the best they can. So don’t micromanage if you’re a tech lead would be my final piece of advice.

Henry Suryawirawan: Really beautiful advice, right? So I particularly like the first one, worry about the trend, right? The direction that you’re going, right? Sometimes you can take some more gradual steps rather than a big bang approach, right? But still, if you’re going to the right direction, I think that’s the important thing.

So Valentina, if people love this conversation, they wanna check out more your resources, is there a place where they can find you online?

Valentina Servile: Yeah. So I’m mostly posting on LinkedIn, so you can follow me there, Valentina Servile, feel free to write. I used to be on Twitter slash X but now I’ve left, so I migrated to Bluesky. You can find me there, but I don’t post as much as on LinkedIn. So yeah, mainly LinkedIn, maybe Bluesky or just drop me an email. You’ll find the address in the book.

Henry Suryawirawan: Thank you so much for your time. I’m sure people learn a lot about continuous deployment today. So hopefully people get to adopt these practices more and more.

Valentina Servile: Thank you.

– End –