#168 - Serverless as a Game Changer - Joseph Emison

[00:01:04] Introduction

Henry Suryawirawan: Hello, Joe. It’s great to have you here. Welcome to the Tech Lead Journal podcast.

Joseph Emison: Great. Thanks for having me.

[00:01:09] Career Journey

Henry Suryawirawan: So Joe, I always love to start my conversation by asking my guest to actually share a little bit more about yourself. Maybe if you can share any highlights or turning points that you think we all can learn from.

Joseph Emison: I’d be happy to. You know, I think my main journey started as a tech lead. And I think the main thing I’ve learned across my career is how important it is to understand the business goals and the business itself that I’m working with or nonprofit organization or whatever. I think when I started, I really wanted to work on cool technical things. And that was my focus. And you know, I realized over time that the quality of the business and the quality of the organization and the quality of the leadership and the culture really mattered a lot more than how cool the technical problem was. And so I’ve kind of developed a rubric for myself that it’s much more fun and rewarding to work with people you like working with in an organization that’s growing. Because an organization that’s growing is creating more wealth, essentially, in many different ways that can be shared. And so it’s an organization that there’s like a healthiness to it that makes it a lot more pleasant to work for. And that, that’s nothing that I optimized for at the beginning of my career. I, I’m on my sixth, starting my sixth company that I’ve been with now for about five years. But every company along the way, I’ve made mistakes around not thinking deeply enough about how growing and how great is this garden that I’m working in, as opposed to, you know, how cool does it seem. That’s the primary piece of advice I’d love to go back and give myself if I could from earlier in my career.

Yeah, thanks for reminding us as well about choosing the right place to work with. And it’s not all about cool technologies, right? I think there are still some of us technologists who love to play with the new techs, right? Especially these days there are a lot of advancements, new things, all the time. So always understand about the business, the quality of the business, the people you work with, the culture. I think that’s also important. And yeah, working in a growing organization I think that’s also very important.

Joseph Emison: No, I was just gonna say, you know, if you pick a growing organization, there’s going to be plenty of money and advancement and things for everyone. And so that one I think is a real key. It is a way to get everything. You can have a good culture and it’s not growing and it’s not gonna create those benefits for you, but if it’s growing it will.

Henry Suryawirawan: Yeah. Not to mention, when the organization grows, there’s so many challenges you can learn from. I mean, like it’s a double-edged sword. Yeah. lot more challenges means a lot more headaches, probably. But also at the same time, you grow much, much better as a person, maybe, in terms of skillset.

Joseph Emison: Yep.

[00:04:25] Writing “Serverless as a Game Changer”

Henry Suryawirawan: So you wrote a book, “Serverless as a Game Changer: How to Get the Most Out of The Cloud”. So maybe before we start talking about your book, tell us a little bit more your relationship with serverless. How did you bump into serverless? And yeah, what made you started to write this book?

Joseph Emison: Yeah. You know, I’ve been on a journey, I think my entire tech career of trying to leverage the benefits of advancements in technology. And one of the things that I’ve realized over the past 25 years is that we get really big, important advancements in how to build software about every 18 months. Now every 18 months, it’s not so much better that you should like throw away everything you’re working on and like just change everything. But there’s a compounding effect to this, like every 18 months, big changes. And this really hit me hard in about 2007. I had been building software for more than 10 years at that point in time. And we had someone, I had started a company. And we had a company looking at making an investment in us. And they sent someone to do technical due diligence on us and was looking at what we had done. And he said, well, you know, what are you doing with continuous integration? And we were like, I don’t know what continuous, what are you talking about? And he recommended the book, the sort of the key book on continuous integration to us. And it really changed everything in terms of how I thought about not only like, wow, we need to do this. But also, wow, I had no way of knowing this, like in the way I had been practicing. And so at that time, I started asking how do I learn about new things? How do I go on my own continuing education journey?

And that led me pretty quickly to a hypothesis of, we continue to figure out what parts of building software are undifferentiated for us, and we continue to be able to hand those to other companies. And pay them and really pay them per use. Even in 2007, this was true. I mean, 2007 we had S3. Amazon S3 has a great wonderful way to just store data. That it would stay forever. It was very cheap. I mean relative to anything else at that point in time. And when EC2 came out the next year, we had a company that really needed not so much to dynamically scale, but we had a scheduling problem. We would get all this data in and need to process it and being able to like flex out to N number of virtual machines and then close them off when running it was a perfect need that we had. Then Hadoop came out and we saw, wow, then this, all of this stuff we were building ourselves, we could actually like outsource that even a bit more. Elastic MapReduce was Amazon service at that point in time.

And so I saw this trajectory of I can take these parts of these programs that we’re making or the workloads that we’re running and processing for our companies, and we can hand them more and more off to people and we can break them into chunks. I mean, from the beginning was let’s break the application into pieces and then use services for the pieces. And so when Lambda came out, I think in 2014, it was just another progression to me of how do I break an application up and just not have to worry about running things. And I was already, at that point in time, I had started four companies. And the amount of time and pain that we had to spend as a fairly small staff just keeping things up. I mean we used RDS early. But like we had cases where the volumes filled up and it took us down. And so there was never a case where those services were fully like I can sort of just delegate the running of these. And so really feeling like I can delegate my uptime to a managed service.

You know, starting in 2014, 2015, I was like, this is it, this is perfect. But, you know, interesting. Immediately what I saw was people, in my opinion, designing things poorly. Like looking at Lambda and saying, oh, we’re gonna like put every function at different Lambda and they’re gonna call different Lambdas. And we’re gonna like take an application, which is here, and we’re gonna like, spread all that complexity out across like network latency calls to all these Lambdas.

And so I started talking, so I gave a talk at the first Serverless Conf, I think it was in 2016, 2015 - 2016 hosted by A Cloud Guru. And started writing more and started arguing more about not only like this is how we should build in serverless, also building serverlessly on Firebase and even building applications like on Google Sheets and Google Apps Script. Really trying everything. And interacting with a lot of skeptical developers who would say, oh, that’s nice. You get to build toy applications that are very simple. The real developers don’t use serverless. It doesn’t work. It’s too expensive.

And so I started this insurance company. Branch is a full stack insurance company. We bear the risks, sell home, auto renters, condo, umbrella insurance. We’re the fastest to be able to sell those bundles by orders of magnitude over any other carrier. And I was in all these conversations with people saying, you only know how to build toys. Serverless is just a toy. And so I said, okay. I think I need to address all of those complaints and skepticisms about building serverlessly. And really write down now like no, we built a full stack in a financial services carrier that has lots of regulatory compliance requirements on it. We built it fairly serverlessly. There are no VMs or no containers. There’s no Kubernetes anywhere. And so how do we think about it? How do we do it? And why does this actually work? Why is it not a toy? And so that has been the arc, but I feel like it’s been this sort of constant attempt to take advantage of we get these new technologies and like they can really change how we develop software and make it cheaper, faster, and better. And so that’s been my focus. And I’m happy to share it in this book. If you’re very skeptical of it, this book is designed for you. It’s not designed to say you believe in serverless, here’s how to do it. It’s a book that’s for the skeptics to help you understand how this actually works.

Henry Suryawirawan: Yeah. Thanks for sharing your interesting story. So actually when I read the book, in preparation of this conversation. I’m not a skeptic of serverless, but I get more intrigued by reading what you are trying to explain in your book, including Branch, right, which I probably will talk more about later, like what makes Branch unique in terms of adoption of serverless.

[00:11:07] Software Development Teams Gap

Henry Suryawirawan: But what piqued my interest in the beginning when you said, right, so technology keeps advancing every 18 months or so, right? And the gap, I think you mentioned in your book, the gap between the best software development teams and average software development teams is getting more enormous. I think it’s related to all these advancement. Maybe if you can pick a little bit like why you think the gap is getting much bigger?

Joseph Emison: Yeah. I mean, I think if you think about like, if there’s like a huge change every 18 months, there’s this compounding effect. And so startups that get to start today get to take advantage. There’s no legacy, right? They’re not relying on any existing technologies. So they can just use the absolute fastest, best way to build something today. And if you started a company like we did five years ago, you’re already gonna be committed to some choices that if you were gonna rebuild it today, you would build it differently and it would be cheaper, faster, and better. You know, if you had all the knowledge that you could have. And so we, you know, we look at that ourselves even.

And so, in the book, I give an example of Instagram when it was acquired had 13 developers and doing about the same thing, roughly, Facebook had hundreds of developers. And then before that, you know, Friendster and MySpace had over a thousand developers. And so there’s, and actually Instagram had 13 employees. I think they might have had like five or six developers.

And if anyone has gone through the process of like how much can one developer do and then how much can four developers do? There are these step points where you lose massive efficiency per developer. I mean, I tend to think of it as kind of like one developer, four developers, 12 developers, 25 developers, 50 developers, a hundred developers. And once you’re at a hundred developers, there’s like a core inefficiency that’s in the whole operation. I mean you’re getting an order of magnitude less productivity than if you have a solo developer.

Now you can’t run everything on a solo developer. It’s not a sustainable thing for a company. But there’s also a reason why Amazon has these like two pizza teams and trying to keep these services small. It’s the only way to keep really good quality and velocity together. And so when I say that the top teams are orders of magnitude better than the average teams, it’s largely that the top teams are leveraging technology so much better than the average teams that they’re just not bogged down. And I feel that very much at Branch.

The company that I was working on 10 years ago, BuildFax, which was fully in the cloud, infrastructure as code, but like running on VMs. I think it still probably is, you know, in Amazon load balancing. We were using RightScale at the time to do some of the orchestration, there was a core inefficiency that we had in running and a core inefficiency that we had in building new features that Branch doesn’t have. Branch is just faster. And so the average developer at Branch is just producing more for Branch and at a higher code quality than the average developer was at BuildFax.

Henry Suryawirawan: Yeah, it’s quite astounding when you mentioned about this Instagram only have maybe 13 employees or five developers in total, right, but can build a world kind of scale, you know, like with users from all around the world. And I think what you say is right. So for people like me, who has been around in industry for quite some time as well, right? I mean, in the recent years, you can actually see that, oh, there are so many technologies that I’m not even aware of, but actually it’s really cool to probably prototype new applications, right? Building something from scratch. And even like, for example, you mentioned, so you can actually build something like only Google or Amazon can do last time, right? And now you can actually tap into those technologies to build similar things like them.

[00:14:52] Differentiated vs Undifferentiated

Henry Suryawirawan: And I think what you mentioned as well, right? If the developer can produce more, that means they’re focusing a lot more on the right thing, right? Which you mentioned as differentiated versus undifferentiated things. I think this concept is very important for you to actually adopt as technology leader or technologist, in general, right? So tell us more about this differentiated versus undifferentiated.

Joseph Emison: Yeah, I think you always wanna look at what does your business need to do differently or your organization need to do differently than other organizations. And what are things that it’s fine for your company or organization to do as well as other companies do, like the best other companies doing it. And so let me give two examples that are both undifferentiated, but that will generally, in my experience, get very different reactions from lead developers. So let’s start with the simple one where I think we’ll all agree.

So if I talk about sending text messages. Sending text messages is undifferentiated. You don’t care if you send text messages as well as the best other company sends text messages. Now, by the way, there’s probably some listeners who are going, no, no, no, no. If you don’t register your 10 DLP campaigns, you know, you’ll get the messages rejected, yeah, yeah, yeah. But if you do it as well as the top people, you are fine, right? If you can send text messages as well as, you know, a quality things, you will be registering 10 DLP campaigns. You know, you’ll have your short codes. And you’ll use Twilio or something like Twilio, but Twilio’s a great product. Uh, I know it’s a little expensive for what it is, but it’s a great product. And so I don’t run into a lot of lead developers who will tell me, no, I need to buy, you know, separate boxes and hook up phone lines and have telephony cards and like run them in a data center. I don’t find that, although I did that in 2005, I ran IVRs that way. So I remember that, it was awful, I would never do it again.

And so I think everybody agrees that sending SMS is undifferentiated heavy lifting and you should use a service like Twilio to do it, right? And in fact, you should probably use a service like Braze or Iterable or Customer.io to like actually trigger those, cause those help make it even easier. And you can do templating and have people do that. And if you don’t know these services, you should look at them up, because they’ll make your life easier. And the back of my book actually has a whole appendix on you should know all these services ‘cause they’re very useful.

Now, let me get to a controversial example. And I don’t know why this is, I mean, I know sort of why this is controversial. Most lead developers want to run Elasticsearch or like Amazon OpenSearch themselves as their search index, when there is a service called Algolia that will do all of the painful stuff for you. I will tell you that running search index that’s performant is undifferentiated. You do not need to do it better than anybody else. I mean, unless you are competing with Algolia as a business. And so you can outsource a lot more to Algolia. It works phenomenally. It’s really fast.

Elasticsearch, OpenSearch, they’re very finicky services. You’ll have to be responsible for their uptime. Indexing to them is kind of a pain in the butt. Generally speaking, frontends can’t talk directly to them, because the security model is too poor. So you’ll have to build a proxy, you’ll have to query through your backend. It’ll slow it down. It adds more development. But most lead developers will say, wow, I looked at the pricing for Algolia. It’s really expensive, and I’m not quite sure, oh wait, how do I do it in a dev environment in the right way to save costs. And no, no, no, it’s just safer, I’ll run it myself. I know how that works. We’ll run these things. We know how to run them.

But like that is classically undifferentiated heavy lifting. Everything you’re choosing to do there to run OpenSearch or Elasticsearch when Algolia exists is undifferentiated. And if you take an actual cost of ownership view, like the time that it costs and the people that are needed internally to run that thing, Algolia is much cheaper. Just period. But most developers will make the choice on behalf of their organization to in-house this undifferentiated heavy lifting with something like Open Search or Elasticsearch because of honestly this misplaced sense of how they should be optimizing. And so that’s my favorite example for this undifferentiated heavy lifting. If you can outsource it, if it’s not something that makes you different, you should use a service, because you’ll always be asked to do more things than you can build as a developer that are differentiated, that are like special to your organization. So don’t add in things that aren’t.

Henry Suryawirawan: Yeah, I like your last line, right? So you are always being asked to do more things that bring differentiators to the business, right? So I think not just Elasticsearch or search kind of a technology, people these days run their own logging, you know, messaging bus, maybe even containers, right, on Kubernetes and all that. So I think that takes a lot of engineers’ hours and also effort. And not to mention like if you wanna make it production scale with high availability, right? That takes even much more effort and actually cost, right?

Joseph Emison: Yep.

[00:19:57] The Real Costs of Software Development

Henry Suryawirawan: Yeah. So speaking about cost, right? I think this is probably where the, I don’t know, misconception or miscalculation, so to speak, right? Like why people are still opting for managing things themselves. They think, oh, it’s open source, I can also run it myself. You know, maybe build POC. It works pretty simple, right? And they just start from there. But I think the most important challenge is to understand the real cost, right? In your book, you have some breakdown of costs. Maybe from your advice, how would you tell us to look at cost differently, so that we can actually see the differentiated versus undifferentiated?

Joseph Emison: I mean, I think that you really need to internally understand how much it costs for you to develop and run things yourself in terms of your people costs. And I think you also need to understand and in terms of your velocity costs. So in general, the more teams that you have in your company that are required to get code live, the more inefficient you are, and it’s a compounding effect. In my book, and everywhere I talk, I recommend people read Accelerate and use the DORA metrics. We use LinearB at Branch as a way of monitoring the DORA metrics. The cycle time is critical. And so, you know, what I see is that the more undifferentiated work you do, the longer your cycle time. And especially when the undifferentiated work is in running the things.

And so when you use a service like Algolia versus you are running, you know, your own Amazon OpenSearch or you’re running Elasticsearch on containers and you’re building a proxy service in there, those are orders of magnitude different impact on cycle time when you release changes to how things go to the index. And I think if you’re honest with yourself both about the actual cost of people and about the impact on something like cycle time or the other DORA metrics, it gets really clear that you just don’t wanna run. I mean, ideally use managed services or, you know, don’t write code, use managed services always gives you better outcomes here.

Now, always does have an asterisks. I mean there are services that are very expensive that don’t make sense for you to use. So like, Algolia isn’t a solution for everyone all the time, because, you know, if you have an index where you’re searching, you know, billions of records, probably, Algolia is not reasonable. But the way I generally think of this is there’s like 90-95% of the time we’re all kind of building the same apps. They don’t have like enormous traffic. By enormous traffic, I mean they don’t have so much traffic that your data transfer costs are like the biggest part of your bill. Generally speaking, you know, none of us have more than, I don’t know, a thousand simultaneous users. I mean, generally speaking most applications don’t have more than like 15 simultaneous users. Truly simultaneous. But, you know, a thousand simultaneous users, most of our database tables at most are sort of in the tens of millions at most. You know, there’s a way to set that up. And so there’s sort of a standard 90-95% of the time.

And yet, most or many lead developers I find, always want in their head to be designing for like huge amount of data transfer, like, you know, millions of simultaneous. Like that’s the model of aspirations, like built this like tank. But the problem is like those just need to sit. Those are all exceptions. And most of the time, actually looking at DORA metrics, actually looking at cost, you’ll find that managed services that are used by everyone at Twilio and Algolia, Cloudinary, whatever, are just gonna beat any sort of honest and fair comparison of actually taking into account the cost.

And I’m always shocked at how many people just don’t want to even think about how much developers cost. They want to view them as like developers are free. And so if you’re a lead developer and you don’t know how much your development team costs, and you’re making decisions based upon this is more expensive than that. Like, you need to find out how much the team costs. I mean, like, if you don’t know, you obviously can’t be making good decisions on what actually costs too much or too little.

Henry Suryawirawan: Yeah. So I think speaking from my experience, what I can see in my area, right? Industry startups. A lot of people, especially during the tough time recently, right? The winter time some people call it, right? So they just look at the bill. They will see, for example, all these managed service or serverless, it costs a lot. The easiest one to reduce is actually, you know, those bill, right? And since we have already a number of people, right? Okay, maybe let’s just switch that to something that we can manage ourselves.

But what you’re saying is true, right? So at the end of the day the cost of the people needs to be calculated equally, right? So you cannot just say, I reduced the managed service bill, but actually that cost is translated to your engineer hours, right? And also probably headache whenever there’s an incident or trying to scale things, right? The on-call thing will create some kind of a cultural issue within the team as well, right? So the harmony might not be as good as before.

[00:24:58] The Serverless Mindset

Henry Suryawirawan: And speaking about these managed bills, sometimes I find as well, the bill is so high, because it’s unoptimized. The usage is unoptimized. So maybe also you can think of just optimizing rather than switching gear to managing yourself. So I think that’s my experience as well.

Joseph Emison: We have these built in moments, right, where every quarter or so we take a look at all the bills that are more than $50,000 a year or $30,000 a year, and we just go ask, you know, what can we do to get them down? We move services a lot. And one of the benefits of serverless is, because you have this really nice container, right? I’m using this service. It’s got a well documented API. I know what I’m using in it. It’s all isolated over here. It actually turns out to be much easier to switch them and move them out than it is if you build it internally. So we’re on our like fourth referral management service. And we used airplane.dev as an automation tool, and they shut down with 60 days notice, I think today is the last day they’re running. And we actually decided to in-house what we had put there, but it was so much easier to do it, because we could just copy their API and so we were able to take all this stuff we’d written for Airplane and port it over. You know, I mean, I don’t want to understate the developers who’ve been working on that have like been working very hard and like week nights and weekends to like get that done.

But to think that we took a service that we were spending like $60,000 a year on and we were able to rebuild parts of it, like a little shim infrastructure for what we were using of it in about in less than 60 days, probably more like 35 days was a result of having used that service, was a result of having this serverless mindset, and of being able to say, okay, now we understand the problem so well and we isolated it so well, that actually rebuilding it or understanding how to shift it somewhere else ended up being much easier.

And so I love being in an environment where everything is kind of contained and small. There’s good separation of concerns. And the best way to do that isn’t microservices. The best way to do that is to use a lot of managed services, because they’ve already built all those great APIs for you. And then if, look, the case of it’s too expensive or I’m locked in is you just build it then. I mean, the thing that’s so wild to me is people saying, like serverless is about locking, it’s terrible. It’s actually like a very cheap way to like learn how to build something and then if you’re gonna build it yourself, just copy the parts of it that you like. It’s actually much faster.

Henry Suryawirawan: Yeah, interesting insights, right? So I think many people are, I mean, do afraid about the lock in and all that. But I think, hey, if you can get started very easily and you can use all the power of the people operating those services, right? So they are very well expert in running those services, right? You can actually just use that, because, yeah, it’ll make you start really, really easily.

[00:27:44] Code is a Liability

Henry Suryawirawan: And I think you mentioned earlier that if you have a choice of writing code versus picking up managed service, you should always choose, you know, managed service. And in your book you mentioned this thing called code is a liability. So I think some people think code is asset, right? Some people think code is the most important thing in your company. So tell us why code is a liability?

Joseph Emison: Yeah, I mean, code is something you have to maintain. And so this idea doesn’t come from me. And so in the book there are some good links here. But you should think about anything that I write, is something there that’s necessary for my organization to work properly. That I have to make sure has proper testing, like doesn’t regress, keeps working. And so I have to maintain all of that. That’s a liability.

You know, the asset is the experience you’re building. And so if I can build something with 10 lines of code or I can build it with a thousand lines of code, I’m gonna be much happier in the long run with the 10 lines of code. It’ll be easier to maintain. It’ll have less bugs. It’ll be easier to test. All of those things will be much better. And so the less code, the better.

At Branch, we have a kind of a waterfall process where when somebody says, hey, can we build this thing? Will you build this thing? Our first question is actually, do we need to build anything at all? And so we do a lot of, why don’t you use this software as a service instead? And if we need to make some sort of integration, we’ll do that. But like, let’s just just go buy a software as a service. That’s step one.

Step two is kind of, can we like down scope it? Because if you buy a software as a service, like you get all this features, those are great. If that won’t work, how do we like take out what you’re asking for and make it really small? One of the things that I’ve found over my career, what generally happens in organizations is people say, I want this thing. And they usually make it really big, because they know that you’re gonna go develop it, and as soon as you put it live, you can’t work on it again for a really long period of time. And so they’re gonna pack everything they can into it, because that’s the only way they know how to get something. If you reorganize, and this is very much in line with the DORA metrics and the Agile Manifesto. If you say, look, two things. One, I’m gonna make you down scope this thing to tiny, but two, I promise we’ll keep working on it immediately, like we won’t stop working on it. It’s not a limited time window, and then we work on other projects for a year. We’re gonna keep working on it as soon as it gets released, but it’s gotta be really small. Then you get all this benefit of the feedback and the cycle, so you just squash down what people want to, like, very tiny, like just helping them a little bit. And then you get it live, and then they can see it, and then they can make that next request. If you train an organization that way, and it’s easier as a founder than as someone who’s inside an organization with other founders.

But I can tell you, you can train an entire organization to think this way. And as long as you deliver on, we will make updates. You come to us, we’ll go get other small updates in as quickly as possible. Everything gets more efficient, because, you know, when you build these big projects, like they’re poorly designed, you put them live, they don’t work, right? We realize like, oh, that was a bad idea, that’s a bad interface, people don’t understand it. Like just build it small and watch what happens and iterate them. So we have this whole process of buy a SaaS, down scope, use a managed service, use open source, everything we can to not write custom code or to write as little custom code as possible.

[00:31:06] Infrastructure as Code

Henry Suryawirawan: Yeah, I think it’s a good principle, thought process for anyone who listen, right? So code is a liability, as a reminder, again, so if you think writing a lot more code. And infrastructure code or configuration also counts as code, right? So yeah, do remind yourself.

Joseph Emison: It does, although. And we don’t have a ton of it but I do think that… and this may be an unpopular opinion, but I do find that domain specific languages for infrastructure like YAML and like CloudFormation. I do in my head put that in a different category, and for this reason. My experience is when we write CloudFormation YAML, and we get it right, we never change it. And so the maintenance cost of CloudFormation YAML is very low. And so I, I don’t really worry about having a lot of that. In contrast, something like Pulumi or CDK that’s like code, I find it gets a lot edits. People keep updating it. It’s got a lot of maintenance. And so it has a different profile to me.

And so I would view CDK infrastructure or Pulumi infrastructure as code as this code that you want to minimize. I actually think of YAML as more like a yarn.lock file or something like that. Like it can get really big, but it’s not a maintenance headache. So I actually don’t find lots of YAML to have the same liability problem, because generally speaking, you get it up and it works and you don’t change it.

[00:32:29] Serverless Definition

Henry Suryawirawan: Interesting. So speaking about serverless, right? I think we have talked a lot about serverless, but I think it’s appropriate to get the definition right. Some people think serverless is, you know, Lambda, some people think it’s a function as a service. So in your view, right, what is your definition of serverless? Because I think from your book, it encompasses more than just a function as a service.

Joseph Emison: Yes, and I give, I think, like four definitions in my book of it. My favorite definition of serverless is, it’s not my uptime. And so it’s functionally about not having to run, that if it goes down and it’s my fault, it has to be like I misconfigured it or I wrote bad code, but, otherwise, I don’t control the uptime. But Ben Kehoe also has, you know, Ben Kehoe’s definition here is I’m only doing differentiated coding, right? Everything that’s undifferentiated, I’m handing off. And then there’s a bunch of stuff about scale to zero and things like that.

If your view of serverless is, and I saw a post on LinkedIn recently, which had somebody saying serverless is expensive, and it’ll send you into ruin, and it’s too complex. And just had all these diagrams of like Lambdas calling Lambdas and using AWS services. And like I tend to agree that those are, I don’t like those infrastructure. I’ve never built one of those. And I think that they’re overly complex. And I don’t think he under, I mean, he’s never read any of kind of the serverless practitioners that I agree with at least about what serverless is.

And so to me, serverless is very much about asking how do I use… Most of our serverless footprint at Branch is managed services. So it is not, I mean, we use Lambda. We use DynamoDB. We use Cognito. We use AWS AppSync. These are amazing services. My book explains how we use them and why they’re so great. And so we do use Lambda, but Lambda isn’t… everything in the application. It’s like where we need to run some backend code, we run it with Lambda, but most of the Lambdas are getting triggered by calls through AppSync. And if you don’t know what AppSync is, like to me it’s like a key part of the infrastructure.

I also have written a lot with Firebase. I think Firebase is also wonderful. I think when writing kind of hobbyist apps, I think Firebase is just easier to work with than AppSync in the Amazon infrastructure. So I like that Firebase infrastructure for simpler applications. But the Amazon one is just much better for like financial services, at least, it’s more robust in a bunch of ways.

But that’s how I think about it is really, am I responsible for uptime? As long as the code functions properly, then that’s serverless. And so I often will tell people, you know, it’s serverless to go build your shopping site on Shopify. Like, that’s a serverless architecture for me. And if you don’t think about it that way, then I don’t think you’re in line with what serverless actually is and how to get the business benefits out of it.

Henry Suryawirawan: Right. And you also mentioned managed service. I think for some people, there is a lot of confusion as well. What do you mean by managed service?

Joseph Emison: I think Twilio is the best example for people on a managed service. It is cloud-based API usually. That you pay and they have a well-documented API, they’ve got a whole operations team. It’s multi-tenant as a service, and you pay them money and they’re just up and they do things for you.

Henry Suryawirawan: Yeah. And I think in your book, if I remember, I also see like for example, if you have to patch something, right? If you have to choose the size, if you have to scale it somehow, right? I think that’s not serverless, right? So…

Joseph Emison: Right, right. And there are these gray areas, right? Because in Lambda, you will pick a size. And so like I view Lambda as serverless. And I would just say in Lambda, just buy the largest size, because you get more CPU with it, at least buy, I don’t know, four gigs. They default to this minimum size that’s terrible, and you shouldn’t use it. But you know, that’s a gray area. But yeah, I also think about it like, yes, if you have to patch it, but also if it runs out of disk space, whose responsibility is it to manage that? That’s a question I like to ask a lot. And so if it’s your responsibility, if you fill up a disk, like it’s not serverless.

[00:36:22] Serverless Security Objections

Henry Suryawirawan: Right. So I think that’s a good reminder, definitely. And of course, when we talk about serverless, there are many skeptics like you mentioned, right? So many objections. In the beginning we talk about cost, lock-in. Some people also think about security, right? Because if let’s say you use managed service, those managed services like a vendor, third party, right? You don’t know them. Your data spreads across different services. What’s your take about all these objections? Maybe tell us how would you actually, you know, advise people?

Joseph Emison: Yeah, the most important thing that you need is good third party risk management in your organization. And I would actually say one of the biggest hindrances to serverless adoption in organizations that really want to is that they have very poor third party risk management programs in their company. And what they have decided to do in the company, maybe not even consciously, but what they’ve decided to do in their company is basically make it really hard to do contracts with other companies as a way of security. That’s not security. That’s just pretending you’re secure.

And I recognize a lot of lead developers may not have full control over this, but it’s useful to know. It’s useful to like have this in your head. Your company should have a robust way for you to say I would like to use this company and to be able to review it in a reasonable amount of time and see if it meets the security guidelines for the information that that service will get. You should have a good way of looking at its historic uptime and evaluating whether you think it’s uptime is gonna be appropriate for your use for it. And then, you know, your finance department and your legal department need to be able to review the contracts and make sure that they have an appropriate DPA, for example, and have other things. And then if they don’t meet those things, you shouldn’t do business with them.

But I’ll tell you, as a regulated US insurance company, there are few companies regulated, you know, as heavily as US insurance companies. We use lots of these services. And it’s not a problem, but it’s about having really good third party risk management. And that third party risk management understanding that they exist to help us do contracts and work with other companies, because that’s a key differentiator in how fast we can build. So getting the company aligned with that is a top down goal of like helping everybody understand we need this function within our company to be able to do these things. But that’s how you do it.

I’ve been in charge of information security at, you know, many organizations that have had a lot of compliance requirements. And every time I run it, and they will say, well, we would never pass compliance with that. I’m just like, no, you just don’t know how to do it. And I guess to quote Twitter, that’s a skill issue. It’s not difficult. But it does require sometimes creativity, and it does require knowing your stuff. You do need to understand like, you know, what is our information security policy? Why do we have these rules? Why do these regulations exist? And you need to be fluent enough in it and have a good Chief Information Security Officer who understands them and cares about productivity. So you don’t have all these things, that can be very difficult, but I can tell you that is so much easier to be secure with serverless and managed services than it is with an internal network.

And so, like we are fully no internal network, fully zero trust at Branch. And the levels that we can achieve and what we are able to do on the budget we have which is very low, is absolutely phenomenal, and would’ve been completely unattainable for me 10 years ago at BuildFax, which honestly has what would be considered like best practices cloud architecture today. I mean it’s full infrastructure as code. It’s got great network rules and great VPCs and everything. But managing that, proving that and everything around that is so much harder. And so you know, didn’t leave a lot of budget for other things. And at Branch, we have managed to make budget for just amazingly high quality security.

Henry Suryawirawan: I’m very amazed when I, you know, read that Branch is an insurance company, highly regulated, and yet, you succeed in adopting all these serverless, right? Because I think security, compliance is always top of mind, especially in fintech or, you know, financial services industry, right? But I think what you mentioned is probably like, it’s more about understanding the compliance itself, right? And be creative in solving it. And maybe you put more proper checks and balances before you actually use those managed services.

[00:40:30] Serverless Uptime Objections

Henry Suryawirawan: And speaking about, you mentioned it’s not our uptime, right? So what happens in incidents where the third party is down, right. I think it’s also one of the most common objection. If those core services are down, then we are also down and we can’t do anything. So what’s your take on this objection?

Joseph Emison: Well, one thing is we run in us-east-1, which is great, because if there are problems in us-east-1, we know a large chunk of the Internet’s going down as well. And generally speaking, we go down less than the rest of the internet. So I’ve been in us-east-1 since 2008, and this is what I found. So I remember the outage in like 2012. There was a Netflix outage there. I just realized that, you know, you get a lot of grace when it’s a us-east-1 problem. So in the last five years, there was only one significant us-east-1 outage. It was a couple hours. It didn’t take us fully down. It was the one that like affected Route 53 DNS where it became clear that that service is bound in us-east-1, and it’s not really a free from us-east-1.

And so, you know, everyone says why are you, would you be in us-east-1? But I’ll tell you like if us-east-1 has problems, like no one’s gonna blame you. Cause every, all the other products aren’t gonna work either. And then outside of that, you know, we basically don’t have outages. We did have one in January 2022. So we’ve been selling insurance since the middle of 2019. And so I can tell you other than that couple hour us-east-1 and this January 2022, where this insurance specific vendor we used, and so there aren’t a lot of other choices. This insurance specific vendor we use went down. Other than that, you know, there are kind of partial, they’re degraded services from time to time. Occasionally, something won’t be available. Like we have to pull motor vehicle records and those come from the states and those services go down like every other day, one of those services down for like 15 or 20 minutes.

And so, you know, you just build a lot of alerting around it. And so our applications are very aware of that. But in terms of the core infrastructure of AppSync, Dynamo, Cognito, Lambda in us-east-1, I don’t think those have really gone down at all ever. Even in that us-east-1 problem, they were all still working. I think Cognito had some issues, but was just slow. So I would say my experience with, it’s not my uptime. I hired a company that is very good at uptime, has resulted in, we basically don’t go down. We have much better uptime than I’ve ever had running, you know, in any of these organizations running my own infrastructure. And so we’re running some of the infrastructure. So I think it’s kind of a non-issue. It is certainly possible.

I will give one sort of caveat here is I did build an application using Auth0 for authentication. And that service went down all the time, in my view. Not for long periods of time, but all the time. And like, I would never, I mean maybe they fixed this, I’m skeptical though after their acquisition and like everything that’s happening with Okta. I would never use Auth0 again, and I would be very cautious in authentication services. But I can personally vouch for Cognito and Firebase Auth having been very reliable services, both of them.

Henry Suryawirawan: Yeah. So when you speak about uptime and reliability, right? When you choose serverless, it’s not like any kind of serverless or managed service, right? You also look at their historical uptime.

Joseph Emison: You have to.

Henry Suryawirawan: Yeah. Can they guarantee a good SLA, right? Is there any compensation that they will give whenever there’s ….

Joseph Emison: Yeah. I tend to look… SLAs are non-compensatory, like they’ll never compensate you. So I tend to like an SLA where it’s punitive to the company. What I like to feel is like, if you go down, it hurts you. And I do think like Amazon views these outages very personally and reputationally, and I think it matters to them quite a lot. And so, they’re never gonna like, compensate me appropriately in the SLA. But I think the historic uptime is the thing to look at, and just to make sure you’re getting an honest look at the historic uptime, because that’s the best way to tell.

We do use some services that are in beta. And, uh, you know, we just do a lot of testing to get comfortable with them. But there are occasionally times we’ll say, yeah, we’re not gonna use that yet. We’re gonna wait another six months or a year. But we use Neon really early, serverless Postgres because we really wanted it. And Amazon’s serverless Postgres are not serverless. They’re terrible. And Neon is a great serverless Postgres database. And so we started using them when they were in private beta. We tested them out. We were like how do we feel? And we just built in some backup caching in case it wasn’t available. And you know, that’s worked out really well. Neon’s been very reliable.

Henry Suryawirawan: Yeah. Apart from the historical uptime, I think one thing that is quite, you know, recent trend, right? People updating postmortem whenever there’s an incident, right? If the company diligently uploads postmortem, and they take it seriously, I think that may also be an indicator that the company serious about their uptime, right?

Joseph Emison: Yeah, it’s another reason why Amazon is so serious about that in a way that Google and Microsoft are not as serious about it. That I just feel a lot safer for production workloads in Amazon. Although, again, I run them in the other clouds as well.

[00:45:22] Branch Development Principle

Henry Suryawirawan: I think many people here would have been intrigued by Branch, right? Let’s talk a little bit about Branch. What makes it unique maybe in terms of like of how you run the development team, how you run the infrastructure? There are a few things that you mentioned in the book. Maybe let’s start with your primary development principle, right? Where you say, when we write code we optimize for maintainability. I think this is very interesting. Maybe can you talk more about it?

Joseph Emison: Yeah, I’m always surprised at how infrequently development teams will say what they’re optimizing for in terms of what they’re doing. Like most teams don’t have this principle. But it’s very useful to know when you’re like looking at code or when you’re sitting down to write something. Like what is my guiding principle on how to write this? My experience is if you write something for a good organization that is gonna live for a while. You know, I wrote code in 1997 that’s still being used widely, in Perl in 1997. And so I think a lot about what is the most important for the company?

And I don’t think it’s about making it speedy, right? I don’t think it’s about other things. I think it is, I want an average developer to be able to understand this and work on this. I think far too many companies are like, we’ve got these 10x ninjas. We write for 10x ninjas. We’re 10x ninjas. We just hire 10x ninjas and all that. First of all, if you have a bunch of people who think they’re 10x ninjas, one, they’re probably not, and two, they’re probably writing unmaintainable garbage. It’s a lot better to have people that have the cultural identity of, like, it’s important for me to write…

I do like the idea of like, I’m writing this, and the next developer has to work on this has a gun to my head. Like, write this thing so it makes sense, so it’s easy to read, so that the code review that you have. One, you should have code review. You know, everyone, not senior developers should have their code reviewed. Everyone should have their code reviewed. And we should have, you know, an understanding that I’m writing this so that somebody else, an average developer, can come along and understand what the heck I was doing. And like, let’s do that. And it makes code reviews easier to know, like, this is what I’m optimizing for. It’s like having in a linter or having style rules, you just like have opinions about things and everything gets easier.

And so that’s our primary principle. And there’s a corollary to that, which is less code in general is more maintainable than more code. And so there’s a lot of debate about like, should you DRY things up? How do you think about repetition? I think you can obviously go overboard with DRY. But I do think in general, you know, it’s very common… We hire a lot of junior developers at Branch. It’s very common for junior developers to have patterns that are too repetitive. They are just much better and much more readable, because it’s just… everybody knows the pattern of like, if i=1 then, you know, return one. If i=2, then return two, or whatever like that.

Obviously that’s just, you shouldn’t, that’s too much repetition. Like I find more junior developers tend to not have these like ways of like, how do I abstract this? Like there’s a pattern that I’m repeating here, how do I abstract that? So in a very normal setting, I find that I do reviews and like somewhat frequently say, hey, you can get rid of some of this repetition. Like, abstract some of this logic. And you know, I think that that’s a key part of the principle is to have a less code but not at the expense of maintainability.

[00:48:39] Hiring Junior Developers

Henry Suryawirawan: Yeah. Speaking about developers, I think this is also unique in Branch, right? You said that you hire more junior developers, and actually you have less, so-called infra developers or engineers, right? So tell us like how do you actually run a good insurance company with mostly junior developers?

Joseph Emison: Yeah. My hypothesis in starting Branch was that, and I had seen this at a small scale, but not at a large scale before, is that if we build serverlessly, then mostly we’re gonna be building interfaces. And so my belief was I could hire frontend developers, and I could basically make them full stack developers as long as we were using the same language, JavaScript, TypeScript on the frontend and the backend. And I thought if it’s not our uptime, if other people are doing the uptime, then I should in theory, just be able to hire frontend developers, make them full stack developers. And then that’s all we would have. And then everyone would just be a developer. And then everybody could work on anything.

They could work on the infrastructure, which is in YAML. They could work on the frontend, they could work on the backend. And we have it all in a monorepo. Oh, and we’d have a mobile app that would also be using JavaScript, React as well. And, you know, everything would be kind of the same. It would all be in the same repository. We would deploy it monolithically. Everyone would have their own isolated Amazon account to deploy it. And so this was the vision from the beginning. And it worked! It worked so well.

And actually, lemme take a step back. So at the beginning I said, I can’t hire a typical senior developer. Because a typical senior dev, I know what’s gonna happen. If I hire a typical lead dev who’s listening to this podcast. If I’d hired them into that, the average one of them would’ve said like, oh, I know how to build. It’s not like this. We need containers. We need to go build some containers. We need to do this thing. We need to do that thing. And my vision would be eroded. And, and I believe in empowering people I hire. I wanna hire someone and I wanna let them do it the way they want to do it. And so I said, I can’t do that. So I said, okay, the safest thing that I can do then is if I hire more junior frontend developers who I feel are capable on the frontend side, like I think I can teach them how I’m thinking about building this. And they won’t know any differently, right? And by the way, it’ll be so empowering, because instead of just being a frontend developer who’s like dependent on other people, they’ll be able to work on everything. They’ll work on infrastructure and everything.

And so we did this and it worked really well. And we said, okay, let’s hire people directly out of bootcamps now instead of like with a little experience. And so we did that and we realized these bootcamps aren’t teaching people anything useful. Like we’re having to like retrain them on everything. I’ve got lots of thoughts about bootcamps. And so then we said, let’s run our own bootcamp, because the bootcamps aren’t useful. And then we ran our own bootcamp. And we did it again. And so, you know, my opinion today is if you set this up correctly and if you hire the right people who don’t have preconceptions about, you know, how things should be done, then we have lots of people who are very happy in this environment.

And by the way, we did a search for a VP Engineering fairly early about 3.5 years ago, and found this wonderful guy, Ivan Herndon, who had been an engineering manager at StockX, but mainly frontend, but like really understood everything and had taught at a bootcamp. And he was like fully bought into like this is a great model, I like this. But it took like eight months to find him. Like, it was just like a long journey. But we had built enough then that we went out and hired more senior developers, and we basically screened for like, who’s gonna like this? You know, and we found senior developers who were like I love this, this is great. Like, this is exactly what I want. Like it’s so easy and I have so much control.

And like, again, when our senior developers are given a feature, they do all of it. They can do 100% of everything. The infrastructure, the frontend, the backend. Like we truly have full stack developers. But they’re full stack developers, because the stack is simple, right? It’s all the same language. It’s all in the same repository, right? It has a monolithic deploy. And so we can make full stack developers if we make the stack simpler. Otherwise, it’s much harder to have full stack developers, cause you need to know too many different technologies. And so it’s worked phenomenally well. We just have developers. There’s no frontend. There’s no backend. There’s no API. There’s no infrastructure people. There’s no ops people, because it’s not our uptime.

Henry Suryawirawan: Yeah, I think it’s really interesting model, right? If people haven’t heard about it, I think this is a very interesting thing that you can learn from Joe’s book, right? So I think how to make it work is something that maybe you need to work on more. Like for example training people, make sure your code is maintainable. But I can imagine, for example, having everything as a serverless, managed service, right? Every developer is empowered to not just code and commit their changes but also bring it up to the production, right? And even operating it in a sense because they can just see it end-to-end.

And many people these days try to build platform engineering, right? Simply because they cannot do self service, right? They cannot make deployment themselves. So I think this is a different kind of perspective how you run the engineering team. So if you use more serverless, I think you probably don’t need so much platform engineering effort, right? So you can do it end-to-end.

And I think there are many other things in the book that you can learn from Branch. Like for example, a few things that I learned, the department head actually is the one who kind of like negotiate contracts with the SaaS, not like the central team. And the other thing is like the engineering lead is the one who configure the cloud or be the DevOps engineer kind of thing, right? So I think that’s also interesting.

[00:54:06] Branch’s Cloud Bill

Henry Suryawirawan: Maybe one thing if you can share, right? People might be intrigued like how much cloud bills actually Branch is running.

Joseph Emison: Yeah, I mean, I think in the book I shared, so we have an Amazon account for every developer and for every environment. And every developer in every environment has a full copy of production. And this is cheap when you use serverless, because everything scales to zero. So like the average developer who’s working at least 40 hours a week doing developments, their environment might cost $4 or $5 a month. Because it’s just not that much usage. But yeah, I think I shared a bill where our entire, all developer environments, all, everything was about a thousand dollars at Amazon for that month. We’re now significantly bigger. So, you know, that production environment is probably more like, the full bill, $7,000 or $8,000 a month right now for Branch. But it’s every environment, every developer, everything including like all of the continuous integration testing that we have. So it’s so much cheaper. I mean at BuildFax in 2015, I think the Amazon monthly bill was probably $30,000 a month and that was doing so much less, honestly, in that environment. But having to run, you know, those VMs and containers.

Henry Suryawirawan: Yeah. Wow. 7-8,000 a month for all environments. That’s really low.

Joseph Emison: All environments, yeah.

Henry Suryawirawan: Yeah. And you are full running insurance company as well. So kudos for you to run that.

[00:55:27] 3 Tech Lead Wisdom

Henry Suryawirawan: So I think we reached the end of our conversation. So before I let you go, so to speak, I have one last question for you, Joe, it’s been a pleasant conversation by the way. So this question I call the three technical leadership wisdom. You can think of it just like advice that you want to give to us to learn from you.

Joseph Emison: Yeah. So I got three here for you and I’ll start with some of the things I’ve said before, which is one, you should have an optimization principle for how to write code. It’s just as important as having linting rules or style rules. You should also have an opinion about what you’re optimizing for. So we optimize for maintainability. I think it would be fine, you know, in some places, it might be we’re optimizing for like the lowest latency execution or whatever it is. But you should have that, you should write it down. Because it will help solve problems and it will help resolve questions and doubts when you’re not there. So you should do that.

And then, you know, my view is optimized for maintainability is a great default. And the second thing is less code is more maintainable than more code. And that is just true outside of like obfuscation contests for like small lines of code. And so you should strive for less code. Again, the book has sort of an interesting waterfall about how do I, what is the tactic when I’m being asked to build something? How do I end up with less code there?

And then finally, I have a saying that I give all the time. So I’ll give that as my final one, which is along these same lines. But it is that it is better to spend two weeks researching and two days developing than it is to spend two days researching and two weeks developing. And I’m always just amazed by organizations that don’t give developers time to research how they would do something. And I’m actually less amazed at developers who just wanna start building. It’s the hardest thing, I know. I mean I always want to start building too. But the more you can train yourself and your teams to think about how do I spend a good period of time where I’m just trying to understand what’s out there? What are the options and how do I make that a good thing? And like, I’m gonna throw away whatever’s done at the end of it. And this really, you need leadership to help you with this, but I think as a tech lead, you can set this tone.

And I’ll give this as the example, our Airplane migration. When we found out essentially like January 5th that we were gonna have 60 days, we actually spent basically all of January trying to figure out what we were gonna do. We looked at a bunch of other services that were options and we evaluated them and we talked to their teams and try to understand things. We looked at what it would take to build it ourselves and what that would look like. And so we spent… I think a lot of people would say you spent too much time, you know, like not acting because this server is gonna shut down and it was critical production functions.

But the reality is you just can’t make the right decision unless you spend time understanding what’s out there. And today, if you’re building new things, one of the things you need to know what’s out there is what are the managed services that could do a big chunk of this for me. And you’re not gonna understand those unless you give yourself the time to play around with them. And almost all of these providers will give you free trials. You may have to talk to someone, and I know a lot of developers really hate scheduling the meeting and doing the Zoom demo and stuff. You may have to do that, I’m sorry. But it’s still worth it. You should do it.

Don’t say I can’t do a self signup on this thing, I’m not even gonna consider it. Or there’s no pricing page, we can’t consider it. Don’t do that. Go understand it even if you don’t use it. There’s actually real important value in understanding what is the service. Get access to documentation, understand what that interface is, understand what that price is. All of that will really help you understand what you need to do better, even if you don’t use it. And we just don’t do that. We don’t do any research and planning in dev. It’s like, oh, I know one way to build that so I’m just gonna do it that way. And like don’t do that. Do take weeks to research. It’s worth it.

Henry Suryawirawan: Yeah, not just research in terms of product but sometimes like design, right? Like how would you design your solution? And sometimes understanding the problem itself, right? Because sometimes…

Joseph Emison: Oh, absolutely. All of those critical. Yeah.

Henry Suryawirawan: Yeah. So thank you so much Joe for this opportunity for this great talk. So if people love this conversation, they would want to connect with you or find more about your resources, your book, is there a place where they can reach out online?

Joseph Emison: Yeah. I’m Joe Emerson on Twitter/X. And yeah, my book’s on Amazon “Serverless as a Game Changer”.

Henry Suryawirawan: I really highly recommend people to read it if you are into serverless. And if you’re skeptics, I think you can also read this book just for your knowledge. So thanks again for your time, Joe.

Joseph Emison: Thank you.

– End –