#198 - Better Software Faster: Measure & Improve Developer Productivity with DX Core 4 - Laura Tacho

[00:01:37] Introduction

Henry Suryawirawan: Hello. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I’m very happy to have another DX person in my show. Laura Tacho is here with me today. So she’s the CTO of GetDX or dx dot com, right? So if people who are following into developer experience, developer productivity, you might have heard or seen DX materials out there, be it research paper or their own products, right? So happy to have you, again, in the show to talk about developer experience and all the stuff that you published recently. So welcome to the show, Laura.

Laura Tacho: Yeah. Thanks so much. Nice to be here.

[00:02:13] Career Turning Points

Henry Suryawirawan: Right. Laura, I always love to invite my guests first to maybe share anything from your career, any turning points that you think we all can learn from that.

Laura Tacho: Yeah. What a big question to start with. A good one. I think there’s really two… I mean, there’s lots of points that have defined my career, but the two that stick out, maybe, interesting to your audience. One was Docker, and getting in, uh, extremely early on a new thing. I think as technologists, it’s really hard to know, like what’s just a new shiny thing that might be a little bit of a distraction and like, what’s a new thing that’s like fundamentally going to change the way that we build, ship, and run software. Even thinking back to like back in 2013, I’m not sure that I fully understood just how big of a difference Docker would make, but to me, it just felt different than like a new JavaScript framework. So not to put down new JavaScript frameworks, but, you know, it just like my intuition, which I think a lot of leaders, we are taught not to follow our intuition, because we shouldn’t act on emotion, we should act on logic.

But truly, what your intuition is, is just hundreds of thousands of data points that you’ve accumulated over your career that allow you to make a judgment about something. So it is, it’s not just fully emotion, but working on developer tools for Docker from very, very early on, getting started with it before it was really heard of anywhere, was hugely transformative to my career. So I guess one lesson, intuition isn’t just emotion, it is data points. So sometimes you have an intuitive feeling about something and you should follow it, it might pay off in a big way.

The second big turning point was like I left my corporate job, a VP job that was like had a lot of responsibility to start my own business and go into coaching and consulting full time. During the pandemic, like on paper didn’t look great. At the time, I was working with my own executive coach and I really realized how much of my career I had let someone else’s career framework decide. And that was really the first time that I said, no, if I want something different, you know, I want more flexibility, I want to be able to go climbing at 11am or whatever it is, I need to fundamentally do something different with my job.

And so I took a very big leap to do that. And it paid off with a ton of hard work. A lot of opportunity and hard work went into creating my own luck to make that successful. I still do coaching on kind of on the side now I’m back sort of in full, you know, full product building, uh, mode with DX. But I think the short lesson there is like don’t let someone else’s career ladder define how you’re going to, you know, your trajectory and your own career growth. You need to, at some point, take responsibility for it and be in the driver’s seat. And it’s surprising how long we all go in our careers without actually being in the driver’s seat.

Henry Suryawirawan: Wow thank you so much for sharing the story. I think those are really great insights, right? So the first thing is about the Docker containerization technology. So I think these days, almost everything is run based on container. Alright, so be it, you know, the CI/CD and all that. So I think that kind of like uh, revolutionized the whole thing, how we get things done. And the second thing about, you know, defining your own career path, I guess, uh, so thanks for sharing that as well. So maybe some people also uh, are looking into, you know, more meaningful career, right? So don’t follow other people’s path. So maybe define on your own.

[00:05:36] Following Your Intuition

Henry Suryawirawan: So, maybe you mentioned something about intuition, right? So the engineering leaders have to kind of like listen to your intuition, you know, gather from past data points probably that you have gathered in your experience or maybe whatever learning things that you have done. So what can you advise us to actually do more in terms of, you know, following our intuitions? Because sometimes, you know, technology change so fast, there are so many things out there, but your intuition is not something that we always, you know, try to sharpen, right? So maybe anything, any tips from you about this?

Laura Tacho: Yeah, you know, this is an exercise and two things can be true. Because on one hand I say, you know, you should trust your intuition. Your intuition is data that you’ve accumulated throughout the course of your career that allow you to make a snap judgment about something or have, you know, a gut feel. It’s not just purely emotional. On the other side, I’m like a extremely data informed decision maker. But I think, you know, if you get your intuition, and actually I think practicing making data informed decisions helps you refine your intuition skills. So changing your mind when new data is available to you, being able to broadly look at lots of data sets and draw a conclusion from them. But not just draw the conclusion, but then know what data you’re going to look for to know that you’re wrong or what data you’re going to look for to know that you’re right.

I think those are all skills to kind of practice that will help you feel more comfortable about intuition. I think the other part of it is like, just like in a recipe, time is important. Like you can’t just throw the ingredients in a pan and like wait a minute. Like there’s a time element of that and I think careers are the same thing. You have to just get the cycles and get the reps in and let, you know, experience will teach you a lot. So time is a factor in career growth. And I think, you know, over time, your intuition gets more honed and more honed. So look out for opportunities where your intuition might be telling you something. Figure out, is there a way that I can use data to see if I’m right or wrong, what data would I look for to do that? And then when you can get some more practice in that, it will just become more natural to you.

Henry Suryawirawan: Yeah, I like that you mentioned time, right? So I think one thing that I would add also like your past mistakes, right? Sometimes, in our career, you cannot always go up, right? So there’ll be ups and downs, maybe situational, you know, change of culture, leadership in the company, or maybe you made the wrong decision to take some career path. But nevertheless, those things are like data points that you can use to you know, hone your intuition.

[00:08:06] Business Oriented Engineering Leaders

Henry Suryawirawan: So maybe let’s go into our main topics of discussion today. So the first thing that I’d like to discuss is something that you actually highlighted before we had this conversation, right? You mentioned one thing that piqued your interest lately is about engineering leaders not necessarily being able to become part of the so called business leaders,right? So, almost in right company, technology is kind of like an enabler, right? Even though if it’s like a technology company, right? It’s like one of the key pillars of the business. But many engineering leaders, after you mentioned that, yeah, I kind of like um, make some kind of like reflection. I think many engineering leaders are not able to kind of like switch their technology mindset into more business oriented. So maybe tell us a little bit more, like what made you come up with this kind of ,you know, thoughts and what do you see from your side there?

Laura Tacho: Yeah, I think right now we’re in, I mean, there’s lots of things going on macro economically, right? We’ve got like the end of the zero interest rate period. We had this like big hiring boom and then a bust and like maybe hiring starting to come back. I think now more than ever engineering leaders are being asked to be more transparent with how work is getting done, how much it’s costing. there’s just like lots of things that before, engineering was really kind of a black box. Like we got away with a lot of stuff, let’s be honest, you know.

I think back to the, you know, like let’s bring this back to Docker. But, you know, I remember the, I mean, the heyday, like the height of the Docker craze when it was just like you, if you knew Docker or knew Kubernetes. Like you could pretty much, you know, no one got fired for learning Kubernetes, right? You know, that’s the saying. You could demand a higher salary. You could do a lot because companies were afraid of losing your talent and having that business knowledge walk out the door. And because of that, there was just, I think, a lot of, I don’t want to say that, you know, there was dancing around engineering. But maybe, you know, we just weren’t expected to show the bottom line for stuff that’s happening in engineering org the same way that marketing, sales, other functions would be expected to. I think that’s really changed.

You know, I spent all my time in developer productivity and engineering metrics, and so I know that that’s the case. I know that engineers are being asked for metrics in a way that they just have not been asked to before, and they’re not prepared to be able to tell that story with numbers. Because now, we have to answer those questions. I think there’s just a heightened sense of accountability from engineering. We’re no longer the, uh, as Charity Majors puts it, the artistes in the organization that we can kind of not work without accountability. But there’s just, there’s more questions that need to be answered now. And I think because it hasn’t been expected of us, a lot of engineering leaders just haven’t had to develop that skill even on the broad level.

But just think about an interaction of engineering wants to do this tech debt project, product manager says we need to build this new feature. And the conversation in a lot of companies is like, well, engineering says that we need to do tech debt. We’re going to do that before we do the feature. I mean, a lot of companies, it’s the other way around. But there was never this, like, let’s actually build out a business case and see what customer impact this tech debt project is going to have. That is something that engineering leaders need to do now in a lot of companies and a lot of circumstances that we just haven’t had to do before. And it’s a really missing skillset.

Henry Suryawirawan: Yeah, you mentioned something about, you know, the situational, about the economic, right? So the end of zero interest rates. So I think in the past, I don’t know how many years, you know, there are a lot of tech startup booms, many engineers being hired, and, you know, the engineering teams keep growing, growing and growing, right?

Now, I think it’s like a questionable kind of a decision, right? If you still want to hire more engineers, right? And that’s why engineering leaders are being asked to actually kind of like be accountable, you mentioned, right? So how you optimize your kind of like resource versus the impact that you bring.

[00:12:01] Explaining Tech Debt

Henry Suryawirawan: So you mentioned about tech debt, right? So it’s always very interesting, right? Whenever engineers talk about tech debt, right? Because obviously the first question is like, how to quantify that?

And I think you also mentioned things like, for example, engineering leaders are not able to explain things in the business sense or business metrics, be it, you know, for example, number of users, number of growth, right? Number of revenue dollar that is being brought into the company. So how do you actually will advise leaders to actually think about you know, switching your mindset, you know, from tech debt, which is probably more like, I don’t know, like complexity or, you know, maintainability and all that into something that is more quantifiable?

Laura Tacho: Yeah. I think there’s kind of two, maybe there’s like two ways to answer that. I think for the, broadly, I want to say just as a general statement that I’m not saying that individual engineers should be thinking about how much revenue that their features are going to bring in, right? That’s not appropriate. What I am talking about, though, is engineering leaders who have a product manager counterpart or like a business stakeholder. Being able to come with the same information that you would expect that business stakeholder to bring about a new feature, come to that conversation with that same information about a tech debt project. So I think a way to get practice here is like, let’s just take tech debt, for example, which is a term that I actually really don’t like. Cause I think it’s too broad. It’s not very descriptive. But if we could pick a, I don’t know, why don’t you pick a hypothetical tech debt project. Can you think of one?

Henry Suryawirawan: The typical one these days, especially if you’re in the big startup, monolith versus microservice kind of thing.

Laura Tacho: Okay, great one. Monolith versus microservice. Like let’s break up this monolith into microservices. Or sometimes what I’m seeing now is like, let’s combine these microservices into a monolith again. Because it’s just, it’s just too complex. Um, But let’s think about that. Like, what’s the actual benefit of doing that? So let’s just say it’s lower complexity whichever way you’re going, if you’re breaking up the monolith or putting things back into a monolith. It’s lower complexity. So what would we expect from an environment where developers have lower complexity code? We would expect that features can be delivered faster. And so we could pick some measurements to measure that. We, we might expect fewer bugs and higher stability, because it’s less complex. We might expect build times to go down, because it’s less complex. We could probably find four or five more examples.

I think for each of those examples, then we can actually tie that to customer impact. So how much does it cost to have an outage? We can look at contracts. If you have an SLA of whatever, however many nines you need, usually if your service goes below that, you need to pay back some of your subscription fee or maybe there’s customer churn, you know, there’s lots of different places that you could look for. What’s the actual impact of incidents and stability problems.

I think the other thing here is opportunity cost. It’s a little bit more difficult to calculate, but let’s say, okay, so every project that used to take two weeks now takes only one and a half weeks. So we’re able to accelerate our delivery by half a week for every project. Over time, that’s a lot of time saved. And of course, we can do the back of the napkin calculation on like salary. You know, just like pure monetary, like how much salary money did we save? But let’s actually talk about what you could have built instead in that time. And what’s the potential economic impact of the stuff that you would build instead?

And so now it becomes a question of like, well, if we wait, we could do this now. And that means that features A, B, and C will be delivered more quickly, but minus the cost of being able to, or the technical debt project, actually the monolith microservice project. Or we could delay it and do it in a quarter from now, which means we know we’re okay with paying that tax, basically, I mean, we’re talking about debt here. We know we’re okay with things moving slower. But the upside of those features going to market first, before we do this tech debt project outweigh the negative parts of waiting for it or whatever.

So, you know, that’s just a kind of simplified example of how we want to start thinking about the business impact, the customer impact, the user impact of something that traditionally has been seen as engineering’s problem and engineering’s kind of self directed project. It’s not. Every single thing that an engineering team works on needs to benefit the business. And your job as an engineering leader is not to say, well, just trust us. We’re in the code all day. It is to figure out what is the way that this actually benefits the business. And it’s just something that we haven’t had to do a lot in the past. So it’s a new skill for a lot of leaders.

[00:16:53] Balancing Between Engineering and Business Focus

Henry Suryawirawan: Yeah. So you mentioned that I’m also guilty in the past, right? So whenever we talk about engineering challenges, engineering project, I think we always kind of like explain it in the engineering complexity manner, and then it stops there, right? We don’t actually translate further into how it actually impacts the business or how can it actually bring more capability into the business, right? So I think, culturally, in the industry, I think, it’s really rare to have an organization that kind of like be able to balance between these two.

And even you mentioned in your, in our pre conversation as well, that you have seen two different extremes. One, maybe organization that focus a lot more on just business impact, you know, being able to churn new features fast, as many features as possible. The other one is actually focusing a lot more on engineering and maybe doing a lot more this, you know, rewriting microservice and all that. So maybe tell us about the danger of these two extremes and how can we actually balance between these two?

Laura Tacho: Yeah, you know, and I think the career change that I went through, and going to be a coach versus so now I get to go extremely broad, not necessarily as deep, but like very, very broad across a lot of companies, which is a huge benefit to do like pattern matching. You know, sometimes I think about myself as like a human regular expression for engineering problems. Like I just, I’m just like figuring out which things belong and which things don’t.

I’ve worked with companies who really have extremely hands off approach to say like, okay, every engineer, you just like, you’re hired to do this job. Here’s the product you work on. Do the thing. And I think what we get in that situation is like a bunch of people moving in different directions. I mean, this is an extreme example, but you know, people are vectors. We have like a speed, but also a direction. And so people can be moving really fast, but if they’re not aligned and going in the same place, the benefit just isn’t going to be as great as it could be.

So that’s on one end of the extreme. I think the other end is I’ve worked with some clients where, I mean, their engineers need to know, not down to the cents, but like how much impact did that bring to the company, you know? I’ve worked with engineers who really like, that’s their gratification centers light up when they know like, oh, great. You know, my whatever optimization I did in, you know, usually it’s more like e-commerce type situations. But like, oh, I optimized this page load. And like, now our shopping cart drop off went from, you know, X percent down to X percent. And that accounts for 1.3 million in revenue and whatever. And I’m like, it’s great that you know that. Like I love that you have access to that.

But I have also some clients that take that to the farthest extreme and say, if we can’t quantify it and if the number isn’t high enough, then we’re not going to do it. And what that leads to is just like a slow degradation of their systems over time. Because, you know, if you cook all day and never do a single load of dishes, it’s not going to take long until it’s just very difficult to operate in the kitchen. And that’s sort of what’s happened.

So, you know, where in the middle is right for your company is really an exercise I will leave to the listener. It’s gotta be somewhere in between the two of those. Like, how can you save space for self-directed engineering projects? With also, you know, alignment and working toward common goals for the business. How can you have both? Cause you definitely can have both. It’s just hard to know. Every business context is a little bit different, but you need to find the place in the middle that’s right for your team and your company.

Henry Suryawirawan: Yeah, definitely. I think every company has a different situational context or maybe time as well, right? There’s a pressure to increase the business impact and there’s a time probably that is more relaxed, so you can do more self-directed engineering projects or maybe equally balanced in any product roadmap. Typically, it’s like, I don’t know, yearly or quarterly, you kind of like right what kind of percentage to work on business related stuff and also the engineering related stuff. I think those are definitely good tips, right? So for people, don’t go to the extreme, right? I think it’s always challenging if you just follow one approach versus the others.

[00:21:00] Building a Case for Improving Developer Experience

Henry Suryawirawan: And I think the other thing that is quite typical these days is about developer productivity and developer experience, right? So I remember in the past when I worked in a bigger scale up, right? So this is always the bigger question to the engineering leaders, like, tell us about, you know, what kind of metrics do you want to use for engineering?

And I think this is always difficult. Especially, if we want to bring up the topic, we want to improve our developer experience. But building a business case is always difficult. So tell us how can we actually build a business case for improving developer experience or developer productivity?

Laura Tacho: Yeah, this is one of my favorite questions to answer because I think that developer experience and developer productivity exist in this very strange area where what is good for your engineers and what is good for your business are the same thing. And there’s so few opportunities as leaders that we get to like do the right thing that’s, you know, the right thing that like we as engineers also want to do, but like that’s also the right thing for the business. Because the bottom line is that poor developer experience just cripples innovation. It makes work and value flow through your company to your users at very slow speed. And reducing friction not just makes engineers more like fulfilled and happy and satisfied with their working environment. It is really good for your customers and for your business as well. And so it makes sense to invest in it. Because if you don’t and your competitors do, there’s only one outcome of that situation, which is that you fall behind. And so developer experience is this interesting thing that is sort of like the overlap of the Venn diagram right there. And I think it’s such, just such an enjoyable problem space to work in.

[00:22:46] DX Core 4

Henry Suryawirawan: Very interesting that you mentioned developer experience is something that, you know, both business and engineers are, you know, incentivized to make it better, right? I think this term is kind of like new in the industry. Maybe, I don’t know, in the past two, three years, probably, uh, it’s getting hotter and hotter, right, in terms of discussions and terms research papers and all that. And more companies are getting into this space as well, trying to understand what is developer experience, right? Because in so many different cultures, different types of companies, experience and productivity can be quantified differently. And I know that you have dealt with this engineering metrics for quite a number of years. So maybe in your point of view, right? So what are the good engineering metrics to use these days?

Laura Tacho: Yeah, well, I can answer that question simply with a new framework that actually Abi Noda and I published along with a collaboration from quite a few other people. So to give a kind of a richer answer, I think I would have said it depends. And my answer would have been it depends for a long time. I’ve been working in developer tooling for over a decade. I’ll, you know, always trying to like make things more efficient, make them better, make them faster, make them more stable or whatever. And like how to quantify this and how to bring that to the business has always been kind of an open question. And because every business is so unique, usually the story that we’ve needed to tell with metrics to make them matter to the business has also been slightly different depending on it.

As you said, the last couple of years, like developer productivity and developer experience are really like having their moment in the spotlight. There’s so many companies that are practicing improving developer experience, like really investing in developer efficiency and effectiveness. So we have just more data to look at more examples. And from all of those examples, we’ve really nailed it down to speed, effectiveness, quality, and business impact as the four pillars to kind of put a story together about developer productivity and developer effectiveness.

So there’s not going to be a single metric. I think that’s an important thing also to call out here. We’ve been looking for a single metric for decades now. Maybe we thought it was lines of code before, and now we, I don’t know, maybe we thought it was number of commits or number of PRs. We’re just not going to get to a single metric because software development is just inherently very complex. There’s a lot of knowledge work. There’s a lot of different processes and social systems that also feed into it. But I think those four really cover comprehensively the most important bits.

[00:25:19] DX Core 4 vs Others (DORA, SPACE, DevEx)

Henry Suryawirawan: Yeah, the new framework that you mentioned is called DX Core 4, right?

Laura Tacho: Yes! Yeah, thank you.

Henry Suryawirawan: Speed, effectiveness, quality, and impact. So it’s very exciting to always hear new research, new kind of metrics, you know, especially, right? So how do you compare this with, you know, the other metrics that were available before? So I think many people would have heard about DORA metrics, right? The four key metrics. And also SPACE framework. And I know DX also published a paper called DevEx, you know, a few years ago. So how would you differ between this DX Core 4 and the other metrics, and is it just another metrics that we have to study and implement in our company?

Laura Tacho: Yeah. So the aim with DX Core 4 is that these, the metrics that we have in the DX Core 4 framework are, you know, like they unify the other frameworks that come before it. And also the research that went into DX Core 4, it’s all building on top of the research from DORA, from SPACE, from the DevEx framework, etc. So this is not meant to necessarily… I think there’s all, often it’s like which one are you going to choose, one or the other, when in fact they sort of all kind of coexist in the same universe and have different purposes. The DX Core 4 has all of the four key DORA metrics integrated in it. It is an implementation of the SPACE framework and it uses the concepts from DevEx. So the Core 4 is sort of everything all together, but simplified enough that it’s easy to get started with.

[00:26:49] The DX Core 4 Dimensions

Henry Suryawirawan: Yeah, so I think that’s a very good thing, right? So now we can consolidate our effort into this just DX Core 4, right? So maybe if you can explain a little bit about those four dimensions, how did you come up with just these four? Are there any kind of research or, you know, something that kind of like brought you into this conclusion?

Laura Tacho: Yeah. I think, fundamentally, the backbone of the DX Core 4 is a concept called oppositional metrics. And you’ll see this also in DORA and in SPACE and in all, you know, many metrics frameworks. But the idea is that when we look at many metrics next to each other, we want to have metrics that are sending signal and working in conjunction with each other that if one goes up, we want to see the other one like also go up. And if the other one goes down, we know it’s a signal that something is wrong.

And so, for example, we could take speed and quality, cause that’s an often the classic trade off is like, are we going to go faster or do we want to make it good in quality? And usually it’s seen as a trade off. And what Core 4 and a lot of these other metrics frameworks suggest is, like it should not be a trade off. You should be able to move faster while increasing quality. And so we want to see those things move together.

The same for effectiveness, which is like the measure of developer experience. How effective can developers be in the system in which they develop software? If we see speed go up, but the developer experience degrade, we know something is wrong. Or if we see a really great developer experience but quality goes down, also not good. So we want to have these sort of levers and checks and balances built into the framework. And so speed, effectiveness, quality, and business impact really cover the biggest corners of software development and keep those things in check.

For the individual metrics, what we’re doing is suggesting a key metric and then several secondary metrics if you want to go a bit further into your measurement. So for speed, that is diffs per engineer. These are like PRs or merge requests or whatever it’s called. This is a metric and it’s how, you know, it’s the first one. It’s the first one in the table. It’s the first one I always talk about, because it is the most controversial, definitely. I think even a few years ago, I would have said like, that’s a bad idea, don’t do it. Um, but we have seen lots of companies use this metric to provide helpful, but limited insight into the health of their software development cycle.

One really important thing about this metric of diffs per engineer is that we’re not trying to see, oh, how many PRs did Henry close, and then how many did Laura close? We’re not looking at that. We just want to look at a developer in your organization. You know, are they closing on average three PRs a week or 10? You know, what is that? What does that look like? Because that can provide us information about how easy it is for developers to develop code, which is like fundamentally what we’re doing here, right? So we want to keep track, like are developer’s able to actually produce code.

That has a lot to do with effectiveness, which is the measure of developer experience in the system. So companies that have a great developer experience have developers that, you know, just don’t lose time to things that are wasteful or friction, you know, a lot of drag. They are satisfied with their tools, their processes, etc. They, you know, they can make room for innovation, those things. So we want to measure that and make sure that that’s, again, one of the four pillars and that it’s keeping balance with the rest.

Quality is change failure rate. That’s a classic good old DORA metric. If you recognize that change failure rate. And then impact is the kind of the business number here, which is percentage of time spent on new capabilities.

[00:30:32] Diffs per Engineer

Henry Suryawirawan: Thanks for sharing this overview about those four dimensions, right? I think definitely the first thing that always uh, kind of like controversial is counting the number of something, right? So this time in the DX Core 4, it suggested that, uh, we should count the number of diffs per engineer, although in the aggregate, not like individuals, right? So this, this diff could be the PR or merge request that we are accustomed to these days. So, tell us why this is something that is more prominent versus the DORA metrics, which previously advocated something like number of deployments, or maybe the lead time, right? Time to bring value to production. So why diffs become more prominent rather than those?

Laura Tacho: Yeah, so DORA metrics are designed to measure software delivery capabilities. It’s really focusing on like once the code is written, how can we get that code out to our customers? So you’ll see things like lead time, deployment frequency, change failure rate, time to recover from a failed deployment, which used to be called MTTR. These are really focusing on that delivery side of things. The diffs per engineer focuses a little bit more on the code authoring side of things, like the, you know, everything that happens before the DORA metrics kick in, more or less. So the activity measurements.

So you mentioned the SPACE framework before. For those listeners who haven’t had a chance to work with the SPACE framework or read about it, the SPACE framework suggests that there’s five main dimensions to developer productivity. We’ve got satisfaction and wellbeing, performance, activity, communication, and collaboration, and efficiency and flow. Those activity metrics are usually what we think about when we think of developer productivity, like commits, lines of code–it’s not a great metric. Other things like that, number of PRs, for example, those all fit into the activity side of things.

We’re not saying like we should ignore those activity metrics. That’s not what SPACE came out to say. They’re saying activity metrics exist. They provide limited insight into how well the system is performing. Also, you need to make sure that there’s all these other things accounted for as well. And so that’s what we’ve tried to do with the Core 4 is to not ignore the activity metrics. It’s still an important part of system performance, but give it more context so that it’s actually meaningful and gives us information. Which is why diffs per engineer, not in an individual level, but not as the only number. We never want to look at that number on its own. We always want to have it accompanied by effectiveness, quality, and impact.

Henry Suryawirawan: Yeah, also, not to mention that you not just have the key metrics, right? The one metric for each dimension, right? You actually also have secondary metrics, which I think in this speed dimension, right, you also have like lead time and also a number of deployments as well. So I think, again, like coming back to what you said, there’s no one single metric. So it’s always kind of like a combination of a couple of them.

[00:33:27] Impact Dimension

Henry Suryawirawan: And the other thing that I find also interesting is about impact, right? You’re not actually measuring in terms of dollar revenue or number of user or number of whatever, right? But actually, it’s kind of like the number of new capabilities and kind of like new innovations, probably. So why is that more important rather than you know, the BAU stuff, right? Bringing more revenue and profit to the company.

Laura Tacho: Yeah. Part of this is like every business is different. And I guess like when we get into it depends territory. Like we are not able to provide outstanding benchmarks about like what is a great business performance for your particular company in terms of like monthly active users or daily active users? That’s not really something. But what we can do is look at, in general, what’s the ratio of like what’s revenue per engineer, for example. For every dollar of revenue that we get in, how much are we spending on developer salaries? That’s something that we can kind of look at standardized and see and compare, you know, companies or like R&D as percentage of revenue.

We can look at different ratios and see, okay, how do we, like, how does this look across the industry? But then more importantly is like, how do we look compared to ourselves? And is this number going higher or lower? We’re not suggesting that you shouldn’t measure daily active users or monthly active users. Like, please do that. If that’s the, if that’s the metric that you’re looking for. But when it comes to kind of broadly speaking about business impact and how organizations are approaching this across the industry, these are a bit more easy. Like they’re a bit easier and more straightforward to standardize on.

[00:34:59] Measuring DX Core 4

Henry Suryawirawan: You mentioned about the it depends answer, right? I think this is quite typical in any kind of discussions about of experience. I even have a friend telling me that, hey, in your past episodes, why is it always kind of like it depends, right? So I think it’s a more opinionated answer is something that many of us uh, would kind of like, appreciate or kind of like want to follow, right? But in this case, DX Core 4, I think it’s kind of like a new kind of like a new paradigms. How can we actually start measuring this maybe in our engineering team or engineering organization? Is there some kind of a different mechanism that you would suggest?

Laura Tacho: Yeah. So in the DX Core 4, we’ll go over the key metrics and secondary metrics, but we also have a third row, I guess, or column, depending on how you’re looking at it. In that table, which is about data collection. So I’ll just like, I’ll get right to my sort of opinionated answer. Everything in here can be collected via a survey. So surveys are super powerful. They’re not just about gathering people’s opinions. You can gather quantitative data via a survey. It’s self reported data, which quite honestly, is high enough fidelity to make the decisions that you need to make with the data. So whether your lead time is 46 minutes or if it’s between 30 minutes and an hour, usually doesn’t matter when it comes to what do we need to focus on and what do we want improve.

So you can gather all of these metrics with a survey and self reported data. You might have to ask some different people for different surveys, especially for different survey answers. Especially when you get into the business impact stuff, because individual engineers probably don’t have information about R&D as percentage of revenue. But they can tell you how much time they spent on maintenance work versus new features last week, you know?

So my opinionated answer is start with a survey. You can get all of this via a survey. And then figure out what would be useful if the collection was automated via, you know, scraping data from an API or integrating with some tool. And then go build that. But don’t delay getting a baseline because a baseline is going to give you direction into what you should start improving and then help you measure your progress. And so delaying a baseline to build something that you’re not even, you don’t even know if it’s going to be useful because you haven’t gone through the process of getting that information, making a decision, changing something, measuring again, to me is a bit of a waste of time. So I would start first with the baseline, make your changes, measure them, and then figure out what you want to invest in when it comes to automated collection.

Henry Suryawirawan: Yeah, getting the baseline. I agree. I think it’s very important as well because in the past I’ve used DX as well, right? So we were actually having a lot of challenges, right? We know we have problems like anecdotally, we can hear from engineers, okay, we had build issue, build time issue, you know, quality issue. But it’s all about opinions, right, rather than kind of like, uh, you know, quantifiable, right? So I think getting the baseline across different teams, across the whole engineering, in fact, I think it’s something that is super critical, especially if you’re a leader, right? Because without that baseline, I don’t think you would know what kind of things you should improve.

And plus, there are so many different dimensions and problems, right? I think it’s also difficult to kind of like improve everything, right? You’ve got to, you know, put focus on some certain areas rather than, you know, throwing all your effort into all different dimensions, right? I think getting the baseline is definitely super critical.

[00:38:19] Developer Experience Index (DXI)

Henry Suryawirawan: And I think another thing that is one of the dimensions that I would like to ask you is about effectiveness, right? So I think in your DX Core 4, right, you mentioned about Developer Experience Index or DXI. Maybe this is something new for some of us who haven’t heard about it before. Tell us, what is this DXI?

Laura Tacho: Yeah. So DXI is a Developer Experience Index. It is a predictive benchmark of developer experience. And all that it is, is just a mean average of 25 developer experience drivers that you can find online. It’s like the DX 25. You can find it on DX’s site. What we found over time now that we have a lot of companies using DX, a lot more data is that companies that score higher in these DevEx drivers have less time wasted. And so that, again, that time waste is not just in terms of like salary and, you know, frustration. But really the main benefit there is like what would you have built instead? What could you have built instead?

Because even if like, so for example, the average developer wastes 20% of their time each week on inefficient processes. Meetings that are not useful to them, waiting around, waiting for feedback, you name it. I mean, if you’ve worked as a developer, it probably, probably doesn’t take you long to list out all the ways that your time gets wasted in a week or when you’re just kind of spent, you’re just waiting for stuff. Even taking that down to 19%, I mean, this is like a minuscule, but like let’s give developers back just like a little bit of time every day. Or like don’t take them out of their flow a little bit more. It does not take long for that to add up to some huge amount of time, of money, but also opportunity. Like what else could you build instead? And so that’s really the idea of this Developer Experience Index. Like it’s not about ping pong and beer, which is often, I think developer experience has a, like quite a marketing problem. I don’t know if there’s a, we should name it something better. But you know, when Abi has this thing that just like makes me laugh every time he’s like, can you imagine if a new salesperson or a new sales leader, like a new CRO comes in and says, well, we’re going to measure the impact of our sales organization based on sales rep experience. Like no way, like you would know, like, there’s just no way that that would ever go over.

And so I, sometimes I feel like, you know, when developer leaders, CTOs, VPs, directors, when we talk about developer experience, the perception is a little bit like, oh, these whiny engineers want to play ping pong and drink beer after 3.30 pm every day. And like, that’s not what this is. This is not a ping pong and beer problem. This is like a crippling, um, you know, this, this stops innovation and it’s gonna cripple your company’s ability to innovate if you don’t fix it. And so putting it front and center in the Core 4 along with the other things sort of emphasizes like this isn’t just a ping pong and beer. It’s truly like a fundamental business problem that needs to be solved. But the DXI, I guess to go back to your original question, is just a mean average of these developer experience drivers that are correlated with lower time loss, better performance in engineering, generally.

Henry Suryawirawan: I think it’s very funny when you mention that, right? Yeah, I think it speaks true to the situation, right? So why there is a developer experience, but not, you know, other departments experience, I don’t know, like sales experience, operation experience and things like that. So I think, uh, it’s very important that we kind of put perspective on this, right? I think productivity is kind of like general term, right? to kind of like measure or improve.

[00:41:54] Impact of Implementing DX Core 4

Henry Suryawirawan: So another thing that I would like to ask you, since you’ve published this paper, right? Has it been implemented in maybe some of your customers or maybe other organizations, be it big or small? Is there any kind of impact? So once somebody has, you know, used this framework, new framework, right? Is there any impact into how they improve their developer experience?

Laura Tacho: Yeah. There’s quite like hundreds of companies using the DX Core 4 right now. And I think like when you start using it, you shouldn’t expect that, okay, now all of a sudden developer experience is going to improve. What it does is it shows you or gives you some models to figure out where do I want to focus my efforts and where should I prioritize our resources in order to fix problems to see the impact that we want to see? The other thing that DX Core 4 really does for an individual engineering leader is it just gives you better set of vocabulary words, common language to take the problem of developer experience and the problem of developer productivity, quantify it, but also tell a better story so that you can walk into your exec team meeting or whatever meeting that you have to go to and draw a line in between developer experience and Innovation capability, for example, in that impact category.

I think, before, with other metrics like DORA, DORA is really great, but it only measures software delivery capabilities. It’s not tying it all together and really bringing it back to the business, which is what we’ve designed Core 4 to do so that an individual engineering leader, whether you’re a manager or a VP or CTO, you can have better conversations about why productivity problems or, you know, lack of investment in things like tech debt or, whatever it might be, why it’s bad for the business. And then specifically, when you change something, what are the results that you should expect to see? And then hold yourself accountable to that prediction.

Henry Suryawirawan: Yeah, so when you mentioned about, you know, not expecting sudden, you know, improvement, right? So I remember in the past when I, you know, used to do this as well, right? So uh, after, I don’t know, like two quarters, uh, we only saw like a mini improvements in some of the numbers, right? So I think some of these, problems are very complex, especially the socio dynamic aspect.

And plus, you know, people change, you know, direction or maybe the mission of the organizations also change, right? And I think uh, don’t forget about that aspect as well. It’s not like something you just quantify and, you know, you put a lot more effort and it will be improved after a while.

[00:44:26] Best Strategy to Improve Developer Experience

Henry Suryawirawan: So I think the other aspect is how do you actually, motivate engineering leaders to actually um, try bringing business case for improving developer experience. And do you suggest also doing it quarterly or is it something like a yearly thing or is there any kind of a best practice that you would suggest us to do?

Laura Tacho: Yeah, I think always, all the time. I think, you know, I don’t have a great answer to the cadence part of it. But let’s take a, maybe, a bigger picture answer, which is like who are your enemies and who are your allies? So I know we’re going to talk about like my tech leadership wisdom at the end of this episode, but like, you know, adjacent to that answer is like, there’s always going to be someone who benefits from an inefficient system. Doesn’t matter how bad something is, there’s always going to be a group or an individual that’s going to benefit from it.

And so you really need to figure out like, why is the systems, like, why do we tolerate developer experience as it is? Or like, why do we tolerate poor productivity or like bad tooling? Like who is it benefiting? Is it benefiting us because we don’t like change and it’s just easier to not change than to go through the discomfort of having to change, you know? Why is that? But try to figure out like, what are the organizational factors that are keeping the status quo in place? So that’s like your enemies kind of category.

There’s also your allies, and those are like your co-conspirators, people who are going to you know, just as fired up about developer productivity and about developer experience as you are. That might be another engineering manager or leader. It’s really great if it can be, like, a product manager or someone from a cross functional function. Or a senior individual contributor, like staff level, principal level. Like, those are really great allies to have. Because then instead of it being you going to your VP or your CTO and saying, hey, we want to do X, it’s like a bunch of you. And there is strength in numbers. There is strength in numbers.

I would also look for the person at your organization who’s going to sponsor this. So like an exec who is going to pave a path for you. You know, you might need more budget, you might need resources or headcount, you’re also going to need sort of like a mandate to be able to get everyone on board with these types of projects being on the roadmap, so to speak. And so who’s going to sponsor you? And then, concretely, what are, you know, if you do establish this sort of like working group, like, what are you really responsible for? What are your KPIs, so to speak? And that’s really where the Core 4 comes in, which is like, how do you measure the impact of what you’re going to do?

So that’s where I would encourage if an engineering leader is like interested in pursuing this is like, first, figure out like your ta…, you know, like your tactics on the ground. Like who are you working with? What are the factors working against you? What are those objections? Who is an executive sponsor that you can, you know, bring this up to and try to kind of make a plan and then pitch it that way, and then take it from there.

Henry Suryawirawan: Very interesting advice you have there. You know, figure out your allies and kind of like your detractors, right? So the one who benefited from the inefficiencies or maybe status quo, right? Sometimes, we don’t like changes, especially if it’s too drastic. You know, coming back to the example of monolith microservice or the other way around, right? It’s kind of like a big change, right? It’s something that not everyone is incentivized to make it successful or make it happen, right? So I think figuring out allies and work together in order to come up with a business case, definitely is a very, very crucial.

[00:47:52] Internal Developer Platform & AI

Henry Suryawirawan: Another important thing that typically is always discussed when we talk about developer experience lately is about introduction of internal developer platform or even AI these days. So maybe in your view, do you have any take of these two, right? IDP and AI, how do you see them as a core components these days to improve developer experience? Or is it something just nice to have for some organizations?

Laura Tacho: Yeah. I think developers see it as a core component. And so what I’m interestingly seeing, I’ve had quite a few conversations about this in the last few months is like, how do I as an engineering leader make a business case for investing in an IDP or investing in Copilot, for example? Especially with AI, some of the promises are very lofty. Let’s just keep it that, like, let’s just say that diplomatically. I mean, you know, I don’t know who really believes that, but like AI, in general, you know, is just sold as like, oh, it’s going to make everyone twice as productive and you can reduce half your staff or, you know, whatever, whatever the promises. That’s just not really what’s happening. It does reduce cognitive load and make developers work faster, help them work faster. But, you know, it’s still, it’s a tool that needs to be operated by a skilled individual and using it irresponsibly can lead to more problems when it comes to maintainability and creating more problems for yourself. So figuring out what purpose these tools should serve and then figuring out a way to measure that impact and tell a story with numbers and make a business case for it, again, is a very important thing.

So, you know, one thing we haven’t really touched on is like attrition or being able to attract talent. A lot of developers, if you ask them, like if they were interviewing for two similar companies and one of them had a Copilot license for them and the other one didn’t, most developers are going to go work for the company that has a Copilot license for them. And so what does that actually cost you? And you can figure out the numbers, like you might have to ask some other people, but like, how long does it take the average from job requisition to be posted online for that role to be filled? Then how long does hire…, like training take, like onboarding? You can figure out how much expensive that is, and then how expensive is it when someone quits and goes works for someone else because they’re not happy with developer experience?

Like we can put a number on attrition pretty easily. It usually costs like 75,000 US dollars, I would say, at a minimum, to replace an engineer. Like that’s mid level or above. And that’s not nothing, you know? That’s pretty significant when it comes to like what are you going to pay for Copilot licenses. But these are some of the ways that I might put together a narrative around something like Copilot or, or an IDP. It’s not just about like, what’s the tangible benefit in terms of productivity gains, but also let’s look at SPACE. What about satisfaction and wellbeing? What about communication and collaboration? What are those other things that are positively impacted by these tools being in place?

Henry Suryawirawan: Yeah, thanks for highlighting that we have to understand the purpose before we implement those kind of a thing. Because many of us, we read from the research out there, people raving about, you know, IDP or AI, right? And we just follow, right? Implement it without actually understanding the impact or the purpose or the even the things that actually sometimes like… in the past episode, I have this Andrew Boyagi, right? So he highlighted the State of Developer Experience report. So he highlighted that leaders are just following, you know, maybe the trends, right? Implement AI and thought by doing that they actually can improve developers. But developers thought differently. They just find like maybe a few percentage improvement, but it doesn’t really actually necessarily improve the whole developer experience. So understanding the true purpose behind implementing those, I think definitely is a key.

[00:51:40] The Importance of Talking to the Developers

Henry Suryawirawan: So we discussed a lot of things, you know, maybe is there anything that, unintuitively, uh, you figure it out, but many people are not aware of in terms of developer experience. Is there something that you want to highlight for us who may not have heard before in the industry or in the research these days?

Laura Tacho: I think that maybe the most surprising thing is that your teams just, they know where the problems are. They work in these tools all the time. And I don’t care what metrics framework it is, like if it’s the DX Core 4 or DORA or any combination of metrics aligned to the SPACE categories. Do not expect that capturing workflow data and looking at GitHub data, JIRA, GitLab, whatever, is going to point you to some, like unknown about developer productivity that like suddenly now that I have all this data about commits and story points and number of PRs and how work is flowing through the system, I’m going to be able to find this magic bottleneck that I’ll be able to fix and like unlock productivity for my team. I have never seen that happen. I have worked with literally, literally thousands of engineering leaders at this point. I have never come across a single case where engineering team, let’s just say, has discovered something new based on workflow data.

Or quite honestly, even like survey data. The thing is like your team is in these two…, they’re in the tools all day long, they know where the problems are. But surveys can be so helpful to just tell you how big the problem is. Like we know the problem is there, and then we can put some kind of formality around measuring how big the problem is and if we’re fixing it or not. So I think this myth of like I’m going to implement, I’m going to get some developer productivity dashboards and like suddenly I’m going to be able to forecast and see ahead when my team has productivity problems. Or I’m going to find this like secret bottleneck. Like it just doesn’t exist, doesn’t exist. Your team knows where the problems are. Just believe them because they’re professional adults who do not have incentive to lie to you unless you give them incentive to lie to you. So gaming the system isn’t actually as big of a problem as we think it is as well. That’s another surprising thing.

Henry Suryawirawan: Wow I think thanks for highlighting that, right? So I, I, I laugh when you explain all that because yeah, I think many people try to automate, you know, getting the workflow data, getting metrics out of the box, right? Only to find that if you ask maybe the engineers or the teams, right, you could get the signals faster rather than spending all the effort to actually come up with the dashboard, maybe. And in the end, you kind of like know that’s the, the, like the problem exists from some people who may have shared it with you. So you don’t need to actually spend a lot of effort.

[00:54:18] 3 Tech Lead Wisdom

Henry Suryawirawan: So Laura, it’s been a pleasant conversation. So I think we all learned a lot about improving developer experience. So as we reach the end of our conversation, I have one last question for you. I call this the three technical leadership wisdom. So you can think of it just like advice that you want to give to us. Maybe if there are anything that you want to share with us today.

Laura Tacho: Yeah. I have so many things. I’ll try to keep it to three. On the topic of metrics, I think the saying that like all systems are wrong and some are useful is really important. And so I would encourage anyone listening to have a healthy dose of skepticism for any type of data that you’re looking at, making sure you understand where it’s coming from, understand the purpose of it, all the things around it, because measurements are going to be wrong. Even a broken clock is right twice a day. So just don’t let data push you to incorrect conclusions. I think that’s a very dangerous area in this developer productivity metric space.

Yeah, I think again, as I said before, things work poorly because it sometimes benefits us to have them keeping, like to stay working poorly. Maybe that’s ourselves and we’re the enemy there, because we are resistant to change. And maybe there’s another factor in there. So I would figure out what those things are, you know, you’d really try to debug the system.

And maybe to follow on this like, you know, metrics thing. A lot of us that are, you know, were previously individual contributor engineers are going to try to pattern match or bring a lot of the skills that we had as an engineer. For example, debugging skills. And try to debug our teams and organizations in the same way that we debug like our Kubernetes clusters or whatever. The thing is, like, your nodes can’t talk to you and they don’t have feelings. And talking to people because your teams are made of people isn’t being illogical, it’s actually the correct way to approach measuring a system that’s made up of people. If we fail to recognize that people make up teams and we try to treat them like a Kubernetes cluster, we’re actually missing sort of like half of the whole system.

And I would just say that’s sort of categorically incorrect to approach it that way. So make sure that you’re not forgetting that software teams and software development is a socio technical phenomenon. So we need to have the social aspect and the technical aspect together. Both of them are equally as important. So don’t forget that humans exist. Developers are adults. They’re professionals. They don’t have reason to lie to you. Treat them as adults. Believe them when they tell you something is causing them friction in the deployment, the development process.

Henry Suryawirawan: Wow, that’s really beautiful. I don’t know why when you explain that, right, I always come back to this phrase, you know, pet versus cattle thing, right? I think for engineers, we try to make everything kind of like uniform. Be it with container, Kubernetes cluster, right? And we treat everything like cattle, right? But actually all individuals are different, right? We have our own needs, our own challenges, right? And don’t forget about the people aspect. So I think that’s really beautiful.

So Laura, if people want to follow you or, you know, maybe discuss with you further about some of these topics, is there a place where they can find you online?

Laura Tacho: Yeah, go to lauratacho.com. That is where you’ll find more about me, my blog. I do have a course on developer productivity metrics. I’m going to be running it more frequently, so you can check that out as well. And you can find me on LinkedIn. Just search for Laura Tacho. I think I’m the only one.

Henry Suryawirawan: Right. I’ll put that all in the show notes. Thank you so much for your time, Laura. I think we all can learn a lot about developer experience from you today. So thank you so much for sharing.

Laura Tacho: Wonderful. Thanks for the lovely conversation.

– End –