#112 - Developer Productivity and Experience - Abi Noda

[00:02:07] Episode Introduction

Henry Suryawirawan: Hey, everyone. Welcome to the Tech Lead Journal podcast, the show where you can learn about technical leadership and excellence from my conversations with great thought leaders in the tech industry. If this is your first time listening to Tech Lead Journal, subscribe and follow the show on your podcast app and on LinkedIn, Twitter, and Instagram. And if you’d like to support my journey creating this podcast, please subscribe as a patron at techleadjournal.dev/patron.

My guest for today’s episode is Abi Noda. Abi is the CEO and co-founder of DX, a platform that helps engineering leaders measure and improve developer experience. In this episode, Abi opened our discussion by sharing what developer experience is, why it is becoming such an industry trend nowadays, and the different ways of how it is being implemented in the industry. Abi explained why the traditional metrics normally used to measure developer productivity do not really work and can even provide perverse incentives. Abi then touched on the two popular researches widely known in the industry, i.e. the DORA report and SPACE framework, before then explained how DX is building on top of both researches to provide the measurements and KPIs to measure developer experience and productivity. Towards the end, Abi shared his advice on how we can start investing in improving developer experience, including when to form a dedicated developer experience team and getting the buy-in from company executives.

I really enjoyed my conversation with Abi, and developer experience is also an area that I have much interest in lately, which I believe can be a great area any engineering team can invest in to improve the overall productivity and engagement. If you find this episode useful, please help share it with your friends and colleagues who can also benefit from listening to this episode. I really appreciate your support in sharing and spreading this podcast and the knowledge to more people. Let’s dive into the conversation, after hearing some words from our sponsors.

[00:05:28] Introduction

Henry Suryawirawan: Hello everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today I have with me a guest named Abi Noda. So Abi is the co-founder and CEO of DX, or some people also refer to it as getdx.com. So DX is a company that helps software organizations to measure and improve developer productivity and experience. Previously, Abi also was the founder of Pull Panda, which got acquired by GitHub, and thus he spent maybe about one and a half years in GitHub to become like the product manager looking at developer insights. So as you can tell, Abi’s background is all about developer experience, developer insights, and productivity. So I’m really looking forward to discuss with you today about this topic because it seems to be pretty trendy these days talking about developer productivity and developer experience. So welcome to the show, Abi.

Abi Noda: Thanks so much for having me.

[00:06:22] Career Journey

Henry Suryawirawan: So Abi, I always like to start my conversation by asking the guests to share more about yourself. So maybe if you can share any highlights or turning points in your career.

Abi Noda: Yeah. First of all, so excited to be here. I’m a programmer. That’s what I do by trade, and that’s what I’ve always done since high school. In terms of my career, about seven years ago, I transitioned from just being an individual contributor to being an engineering manager. And I still remember that shift I made. It was so exciting at the time because you go from being so focused on shipping tasks and programming features, and suddenly you’re now responsible for people and the dynamics of a team. And so, I’m a big fan of your podcast and a big fan of sort of the science and art of software leadership.

When I became a software engineering manager, one of the problems I kind of became fascinated by was the problem of how do you measure software development? How do you measure software development teams? The event that sort of prompted that was, I was the CTO of a startup in California, and the CEO came to me one day and said, “Hey, Abi. Our leadership team is going to start having metrics that we review together each month. Can you bring some metrics for engineering to understand the productivity of your team?” I remember my initial reaction was, “Man, I don’t know if there is anything that really is an accurate depiction of what we’re producing.” And, certainly, there was nothing that I felt would be reasonable to also use with my team. I could report something like velocity points, or number of pull requests, or number of features to the rest of executives, but I knew that number meant nothing to the actual developers on my team. And so, that moment has sparked a, now, nearly seven-year journey to try and figure out this problem.

And along the way, I first started a company called Pull Panda, which had a few different products, but primarily was still focused around this problem of measurement. With Pull Panda, the approach we attempted was to use development sort of exhaust data, right? So data from GitHub and Jira tools like that, and compile those into metrics for managers and teams and leaders to measure how they’re doing. That company was acquired by GitHub, where, as you mentioned in intro, I led a similar product within GitHub. GitHub also wanted to bring a product to market that tried to solve this problem. Then along the way, I kind of came to the realization that there was maybe a different way to approach this problem. Perhaps a better way. And that’s really the genesis of DX, the company that I’ve been working on for the last two years and running today.

Henry Suryawirawan: Thanks for sharing your story. I think it’s really fascinating. You have been dwelling with this problem for seven years. So now that you get a chance to find a company working on that, I think it’s really exciting.

[00:09:14] Developer Productivity Industry Trend

Henry Suryawirawan: You mentioned that you are kind of like intrigued by this problem. These days, actually, it is becoming like a trend. Like everyone’s problem, everyone is thinking about it. Especially so many startups, software companies these days, and they want to measure this kind of a developer experience and productivity. So why do you think only these days it becomes like really hot and trendy?

Abi Noda: I actually don’t think it’s that new. I mean, I think people in business have always tried to measure performance. Businesses and public companies have their stock price public to the whole world. Everyone can see how well you’re “performing”. And, of course, if you’re ever inside of a sales organization or a marketing organization, it’s very much driven by targets and numbers and performance.

I think business leaders have been trying to do the same thing for engineering organizations for a long time. Of course, I think at some point in their career as anyone who’s worked in tech has kind of bumped into this, but the most primitive approach to trying to measure was through metrics, like lines of code or velocity points. Just people looked at programmers and said, “Oh, they produce code. Let’s count how much code they produce.” And, of course, everyone who’s ever written a line of code understands why that doesn’t work. But that approach is still pretty common today, actually, surprisingly, even at really big, reputable companies. But I’m sure we’ll get back to kind of more like the progression of how this happens later.

In terms of, I think, why it’s been a hot topic as of late is partly the shift to remote and hybrid working models. So the whole world changed and business leaders naturally wanted to understand how’s this changed our capacity as an engineering organization. I think it’s also that software is just becoming such a more and more important part of every business. It’s not seen as a cost center as much as a, really, the lifeblood of every organization. You know, the key to success is innovation and technology. And so, I think as more companies realize that, and the job market, the market for talent has become more and more competitive, it’s just created more pressure overall in the whole system to understand how well you’re doing, as an organization or even as an individual. So that’s my perspective.

[00:11:28] Developer Experience for Developers

Henry Suryawirawan: I mean, we have been talking from the engineering leaders' point of view. We want to have metrics. We want to have measurement. We want to know how features get delivered and thus impact the business performance, so to speak. But what will be some of the impacts maybe for developers, engineers or the software teams themselves, if they actually put more focus on improving their developer experience? So maybe you can share a little bit on this.

Abi Noda: I think everyone, in any profession, in any trade, you want to feel like you’re good at what you do. You want to feel mastery over your craft, and I think that’s true, very much true of software developers. It’s very much a craft. There’s very much a lifelong pursuit of mastery in it that almost every professional developer sort of embodies.

In terms of developer experience, developer productivity, and measurement, today, it’s not so much about measurement, I think, for a team as just about working in more effective ways. I think just as much as measurement has been kind of a mystery and elusive problem for software organizations, how to actually be good has also been sort of elusive, right? Software organizations are constantly trying to figure out what technology should we use? How should we organize our teams? What kind of processes should we use to deliver software? Should we do Scrum? Should we do SAFe? Should we not do any of those things? Team topologies? Microservices? It’s just always changing.

And so, to boil this down for a team, developer experience is one approach to thinking about engineering excellence and maximizing engineering performance. And developer experience is really about focusing on the day-to-day experience of delivering software on a team. Understanding what is getting in the way? What are the biggest constraints and bottlenecks and frustrations in that process? And improving those to increase the capacity and performance of the individuals and the team as a whole. So that is really what developer experience is about from the standpoint of an individual team.

[00:13:30] Different Names of Developer Experience

Henry Suryawirawan: And if I’m not mistaken, there are so many terms in the industry referring to the same, almost the same thing, I guess, but not really the same. So, developer experience, developer productivity team. Maybe you can share a little bit more. What some of the companies do to solve this problem?

Abi Noda: Yeah, it’s really interesting. This is something I’ve been focusing a lot on through my own podcasts and research and talking to leaders across the industry. For many years, organizations have stood up special teams or groups within the organization that are devoted to making the rest of the organization more excellent. And, in fact, one of the, I think, slightly dated, I should say, names for these teams are engineering excellence groups. Engineering excellence teams. But when you look broadly across the industry, you see a lot of different flavors of this type of work happening.

I think if you had to give it one label across the industry, I would call it enablement. These are teams, departments whose responsibility is to enable teams and developers across the rest of the organization to be more effective and more successful. So when you look at the industry today, I’ll just list some off, you’ll see platform teams, infrastructure teams, developer productivity teams, developer experience teams, engineering excellence teams, software delivery life cycle teams, engineering enablement teams, forgot that one. The list kind of goes on, and what they all share is roughly the same charter. They all exist to help. They all focus inward. They’re not external customer facing. They’re inward facing. And they focus on helping development teams be more effective. Saving them time. Helping them adopt better practices. Supporting them in specific areas of their features or products that they need support in.

The ways in which these different types of teams I described accomplish that goal are a little bit different here and there. But if you had to kind of take a 10,000 foot view on it, I would say some of the teams tend to be more process and culture focused, whereas some teams are more tools and tech focused, and then others are sort of just a blend of both. To bring this full circle, developer experience teams are, I think, the newest kind of evolution of these enablement teams. Developer experience teams have really been born out of platform and developer productivity teams that were traditionally very focused on internal tooling, particularly around builds and tests.

The way in which developer experience teams are different are that instead of just being responsible for a few tools that developer use, they’re responsible for looking at the developer experience holistically across all the different tools they use. Cause, in fact, that’s where a lot of the friction is between the different tools, not within each of the individual tools. So looking at the developer experience holistically and finding ways to improve that, not only through technology and tools but also through best practices, cultural changes, even organizational changes.

Henry Suryawirawan: So if I also can share, when I read your GetDX white paper, there are certain impacts that organizations can get when they invest a lot more in this developer experience. So things like, of course, efficiency. Developers get more productive and more velocity, so to speak. And then also engineers attrition. I think if you probably work in a company that has so bad developer experience, I guess engineers would probably find another place to work at. And, of course, the satisfaction working there. At the end of the day, it’s always about business performance, right? If the engineers are productive, of course, you will drive business performance as well.

[00:17:15] Traditional Metrics

Henry Suryawirawan: So, you mentioned in the beginning about different ways traditional companies used to measure their productivity. Lines of code, number of PR, maybe PR age, number of bugs, number of feature shipped, software velocity. So maybe tell us more. What are some of the problems if we try to measure productivity by using these traditional metrics? And maybe there are other more traditional metrics that you can share as well.

Abi Noda: Yeah. So the “traditional metrics” or metrics that are, I think, most commonly misused would be, really, anything that revolves around like counting widgets of what programmers produce. I use the term widgets because the flaw is in the way leaders view software development as something that produces widgets. In other words, they view it as a sort of factory assembly line. And as a side note, what’s really interesting is a lot of the most popular software engineering metrics, like cycle time, for example, or lead time. Actually, those have been used in manufacturing for many more decades than they’ve been used in software development. So it’s interesting that our discipline has adopted these manufacturing metrics when anyone who’s built software understands that it’s not a repetitive assembly line sort of process, right? It’s a creative process.

So, going back to kind of what the problematic metrics are. So things like lines of code, number of pull requests, velocity points, those are, probably, the most notorious metrics. Velocity points, which really is about counting the number of agile points that come from sort of the agile ceremony of assigning points in the estimation process, suffers a really, really harmful problem in that agile estimation points are a tool for estimation. And therefore, you’re measuring your organization based on how much work they completed based off of the estimates, which may or may not be what actually occurred in actuality. That’s not even the worst part. The worst part is that by incentivizing teams to be sort of graded based on their velocity points, you’re actually incentivizing them to inflate their estimates. And so, you, actually, in this process, completely ruin the actual purpose of velocity points in the first place, which is to provide an honest and useful way to gauge the capacity and forecasts of delivery for an organization.

Lines of code. I mean, we joked about it earlier, but, again, anyone who’s written software understands that writing more code is not better. So therein lies the critical flaw with using lines of code. But in a more nuanced way, it just doesn’t really make sense, because, as any software developer knows, depending on the programming language, you might write more or less lines of code just based on the syntax. And it’s funny. I’ve actually talked to organizations recently who joke and say they’re actually incentivizing people to delete code, not add code. We want less code as software developers and organizations. Like all code is just tech debt waiting to happen. We want less code, not more. So, incentivizing lines of code just creates a completely perverse incentive to actually write poor quality software and unmaintainable code.

So those are some examples with problems with what I would call the traditional notorious developer productivity metrics. And this has been talked about for decades. I mean, there’s a Martin Fowler article called “Cannot Measure Productivity”. I think it’s from like 2002, where he talks about lines of code and how leaders just keep using it even though it doesn’t work. So developers have been telling leaders that these metrics don’t work for a long time. Unfortunately, that still hasn’t stopped a lot of companies and businesses from at least attempting to use them.

Henry Suryawirawan: So, I guess, one of the common challenges why companies still use those numbers is because probably it’s hard to get the quantifiable measurement, right? So mostly maybe anecdotal or perception within the company. Our build is slow. Our build sucks. Or our environment is bad, but nobody really quantifies that. So I guess it’s really hard for leaders, probably, to choose those numbers.

[00:21:16] DORA & SPACE

Henry Suryawirawan: But I guess lately there are so many researches done around these topics, or maybe previously was there as well, but it was just not popular. So there are few things that I can pick up from the industry. First is about DORA, the State of DevOps Report, or DORA reports some people call it, and also SPACE from Microsoft. So tell us more about these two popular research and what is your thought about using some kind of research-based approach to measure productivity?

Abi Noda: Well, first of all, I think we all need to be looking more to research for answers to this problem because it’s so difficult. Not only is it so difficult, there’s a lot of non-research out there. So many vendors selling their magic metric for developer productivity. So many leaders choosing and adopting ill advice metrics for their organizations that harm their organization. So, we all really need to be more prudent in how we think about this problem, and leverage the wonderful research that’s come on in the last few years.

I’m a huge fan of both DORA, the book Accelerate, and the SPACE framework. I worked with Margaret-Anne Storey, who is one of the authors of SPACE and Nicole Forsgren, another author of SPACE, and of course, she was the chief scientist and researcher and author of Accelerate, and the founder of DORA. DORA was a monumental step forward for the industry because what DORA proved was that engineering performance, software delivery performance, actually helps business. And this goes back to the earlier point I made about how in the past software development was seen as a cost center. How much are we spending on IT? Whereas today, we know that all the biggest and most successful companies in the world are really technology companies.

And so, Accelerate, above all else, proves that using real rigorous statistical research. One of their findings was also that there were these four key metrics, which made up the construct, which they called software delivery performance, which correlated highly or predicted business performance. Those four metrics, of course, have sort of taken a life of their own, but Nicole and the other authors of the book never anticipated or intended. We can maybe come back to the merit of those four metrics, but in terms of using them in an organization. The science behind what those metrics correlate to across the industry is real. And Google continues to produce the annual report. Examining those four metrics and stats across the industry to this day.

SPACE, which is much more recent, was about developer productivity, which is very different. Accelerate was about software delivery performance, which is not developer productivity. That’s why Nicole wrote SPACE, cause it was a different problem to be solved. I love SPACE. I love the premise and the findings and the advice to practitioners and leaders that are embodied in SPACE. And I think the biggest shift that SPACE has made in the industry is that leaders are finally realizing that developer productivity is more than just counting widgets. SPACE talks about how there’s five different dimensions of developer productivity and that a developer’s day is complex and multifaceted, and it’s not just about cranking out code, it’s about collaborating with peers and reviewing code and planning and designing features, and even talking to customers. And so, SPACE really put forth the idea that developer productivity is complex and to understand, measure or improve developer productivity, you need to be looking at it from different angles.

And so, you’ve seen organizations try to implement these things, both DORA and SPACE. And I think that kind of brings us along to my work today, which is SPACE and DORA really helped the industry think about software engineering performance and developer productivity in the right ways. But I think there’s still been a real challenge with, okay, what do we actually measure? How do we measure the DORA metrics? Now that we have the DORA metrics, what do we do with them? How do we measure SPACE metrics? What does that mean? That’s really the work that me, as well as Nicole and Margaret-Anne Storey are doing today, is really build on the work that they’ve done. They build on DORA. It builds on SPACE. But what we’re trying to do is kind of bring that into a more applicable set of frameworks and tools and concepts to really help organizations actually apply this. Both measure and improve developer productivity using a set of practices and measures that work.

Henry Suryawirawan: Thanks for sharing this in details. Because yeah, DORA took the industry by storm. I read the Accelerate book as well. It’s really good. For those who haven’t read it, please go and check it out. And SPACE, which came in the last maybe three years or four years, it was also quite revolutionary. Before people were talking about the four key metrics from DORA. But hey, yeah, there’s another angle of developer productivity, satisfaction and things like that, that maybe we should not neglect as well, because that also improves the software development performance at the end of the day.

[00:26:30] DX Measurements

Henry Suryawirawan: I was also excited when I saw that Dr. Nicole and Dr. Margaret-Anne were also part of your company. They are probably the researchers in your company, and hence come with the DX unique way of measuring this developer performance and developer experience. So maybe now it’s time to tell us more how exactly DX actually do the measurement? I know that you have DX KPI numbers, right? And also top drivers that you measure. So tell us more about all these kinds of measurement that DX does differently.

Abi Noda: I think important to start with the philosophy of what we’re doing. Because one of the things that’s been a little lost in all the conversation around metrics all the time is, what are we trying to measure? What are we trying to improve? Again, when you look at some of those old notorious metrics, like lines of code, or even things like cycle time, it’s all about looking at your software developers from the top down, and observing what they’re doing and then making an assessment of whether that is good or bad. That doesn’t actually improve things, and it actually doesn’t even tell you if anything’s wrong, or if something is wrong, what it is.

And so, the philosophy behind developer experience is that we believe that the developer experience, which we talked about earlier, meaning the day-to-day friction that developers encounter, the things that make them stuck, the things that make them frustrated, the obstacles that get in the way from them shipping and being able to deliver things to customers as quickly as possible. Understanding, identifying and improving those things is the key to productivity. So that’s the philosophy of what we’re doing. And so when we talk about what do we measure? It’s all around how do we identify, how do we measure that friction? How do we understand where the friction in the organization exists? How do we know if that friction is good or bad? Meaning, compare it against industry benchmarks to understand is this process slower or faster than other organizations or high-performing organizations? Is this delay typical, or is it a sign of a bottleneck in our process?

With DX, it’s a little bit of a long winded response to get into kind of what we measure. But in principle, what we do is we look at the developer experience as a whole, and we publish the paper that’s on our website. That was our first peer reviewed paper on what actually is developer experience and what is it made up of. So what we do at DX is we identify what are the top drivers of developer experience? What are the things in the day-to-day experience that most affect developers? Things like slow tests or nasty code or poor requirements on the task they’re working on, or delays in handoffs between developers or difficulty building their code, deploying their code, releasing their code, or just slow feedback loops in terms of getting peer review or getting feedback even from customers.

So we constantly are looking at, okay, what are the top things across the industry that affect organizations the most? And then we measure them with a multifaceted approach. So we look at really three dimensions. We look at, first of all, subjective experience. Meaning, what’s developer’s attitudes towards these things? Are they satisfied with their tools? Are they happy with the code review experience? Do they have an easy time to point code or is it really difficult?

In addition to those subjective measures, we also look at objective measures. So we look at, okay, developers feel like it’s difficult to release things. How many steps does it take? How long does it take them to deploy? Or developers are unsatisfied with the code review process. How long do they typically wait to get a code review? How difficult is it to review other people’s work?

So we combine subjective and objective, and the last dimension we measure is around importance. Because something can be really bad, but unimportant because you don’t experience it frequently. Vice versa, a really small, I like the analogy like pebble in your shoes, something really small that’s really not that bad, but you experience it every single day, could be really important.

So we really look at those three dimensions. I like to call it a triad of developer experience. Well, really, you can measure those things in a number of ways. A combination of surveys as well as helping customers instrument their systems to get some of that data in real time. We bring that all together to customers and help them make decisions on where to actually invest to improve their organizations. It was a little bit long winded, but in a gist, that is how we think about improving developer productivity and how we approach measurement around developer experience.

Henry Suryawirawan: Thanks for sharing in details. Actually, it really gives the listeners more concrete ideas of how DX works. And apparently, I also have given it a try. So the way it works, like you ask developers through surveys, and then they will provide answers. These three dimensions, like the subjective experience, the perception, and then the objective part, right? How many, probably, days it takes you to finish a task, for example? And you have both subjective and objective measurement. And the other part, the dimension of importance, is for you to choose after you get all these results, what will be your focus area? So the team themselves can choose. Each team can have different focus area, so to speak. From there, you will then do it again over the time, and hence, creating this, maybe, flywheel effect that you keep on continuing to improve the developer experience in different parts of the organizations.

[00:32:01] DX KPIs

Henry Suryawirawan: And I just want to highlight also this DX KPIs. So all these top drivers and all these results, actually, DX tries to come up with much more like four key metrics in DORA, right? Like four KPIs from DX, which are engagement, productivity, effort to deliver, and time loss. It seems like different from DORA or SPACE. So tell us more about these KPIs.

Abi Noda: Yeah. It’s a great question. With developer experience, it’s about what a KPI is, is like North Star metric. It’s about what is the goal of all this? So when you take a step back and look at an engineering organization and you’re talking about productivity, you’re talking about developer experience, what’s the goal of all of that? And that’s what the DX KPIs are. So as you mentioned, if your developers are really productive, and your processes are really efficient, those map, those drive, those predict will correlate to better performance across the KPIs.

So developer engagement is a measure similar to employee engagement of how energized and excited developers are in their work. And that’s such an important KPI for businesses because if you know developers, their actual day-to-day productivity is very much driven by the level of passion and excitement they have towards what they’re working on.

Perceived productivity looks at a slightly different dimension of more cognitive and rational one of how productive do developers self report themselves to be. I mean, every manager will ask their developer, “How productive did you feel this past week?” And that tells you a lot. That’s essentially what we surface at the organizational level with that KPI.

Effort to deliver really centers more around tools, although it does touch on practices. If you think about the ideal software delivery life cycle from the standpoint of an individual developer, you want it to be easy. You want it to be really easy to start on a new feature, make changes, test those changes. Maybe deploy those changes to another environment, staging environment, manually test them, send them through your CI/CD pipeline and ship it to customers. You want that to be really easy. And so, effort to deliver is around understanding the overall ease or difficulty in that process.

And then lastly, time loss is about literally how much time is being reported to be lost due to waste and inefficiency in the process. So it provides a way to understand those bottlenecks, the things getting in the way of developers in a way that can actually be translated to dollars. And that’s what’s so important about that last KPI. Whereas the first three are similar to employee engagement in that they’re sentiment scores. Time loss is still a perceptual measure. However, it can be conveyed in dollars that matter to the business.

And so, with these four KPIs, it’s really about aligning around what does a good software engineering organization look like? That’s what the KPIs really provide to businesses. In terms of how we came up with the KPIs, they’re actually rooted in our research paper. I won’t go into full length at how that works, but, really, one of the things we know about developer experience or experience in general is that it’s based on this concept in psychology called the trilogy of the mind. And so, there’s really three dimensions of experience, and three of the four KPIs, so all the KPIs except time loss actually map to those different dimensions of experience in psychology. And they’re also useful to businesses.

With our KPIs, one of the things that we’re working on is understanding their predictive power, similar to what DORA did. So we not only believe that these are, from a research standpoint, the right North Stars for organizations. We also believe that these North Star metrics will predict and correlate with better business performance and outcomes. Things like attrition. Things like financial performance or non-commercial performance. And that’s a research we’re still working on and hope to be able to share with the world early next year.

Henry Suryawirawan: Wow. Sounds really exciting. I’m looking forward to another research to compliment DORA and SPACE. I’m sure looking at Dr. Nicole and Margaret-Anne as part of the research team, I’m sure it’s going to be revolutionary as well. But for all DX users, maybe you get a glimpse of it since you’ve been using it as part of the software itself. And I think these KPIs, the most important for leaders, right? Because previously, they want to get numbers, the targets. So these KPIs sort of give them the numbers that they can aspire to improve, and they’re just four. Not many. So you can actually look at the KPIs for leaders, and then for engineers, the top drivers, probably the focus area that you will improve.

[00:36:49] Starting With Developer Experience

Henry Suryawirawan: So I’m sure all the listeners here who are engineers and maybe leaders, they’re excited. They want to improve developer experience, but maybe they don’t know how to start or maybe getting into GetDX is one. But how should people build a case in a company which probably hasn’t invested a lot on developer experience?

Abi Noda: Yeah. So I talk to a lot of companies about their journeys and investing into developer experience, and in fact, I talked to a lot of companies that are really early on that journey. So companies where it’s just a couple of developers, kind of grassroots trying to tackle developer experience, all the way to large corporations where C-level executives are excited about developer experience but don’t know how to actually begin working on it. I think from everything I’ve learned with developer experience, I probably wouldn’t recommend just one day waking up and saying, “We need to do developer experience”. Cause that’s not really what developer experience is about. Although it is now a buzzword. I think it’s just dangerous to kind of chase trends and fads and buzzwords in the tech industry cause that’s how we ended up with like a thousand JavaScript frameworks.

With developer experience, I think the success stories I hear often all revolve around there being an identifiable pain in the organization. That can take many forms. Sometimes it’s that people are leaving. There’s attrition. In fact, that happens at Airbnb. Sometimes it’s that things are just slow. Hiring more and more developers, but you can tell it’s just becoming harder and harder to actually deliver things, and you’re not really seeing things accelerate as much as they should as you hire more people. Other times, you’re well aware of developers complaining about the problems and challenges they’re having in their day-to-day work. Whether it’s the build times going into the hours or even days, or multi-step process to test work before they can deploy it to customers.

And so, I think beginning to tackle developer experience should really just begin with picking one concrete thing within the organization. Something that has leverage. You want something that’s going to provide a return for the business. Simple way to illustrate this would be if builds take, let’s say, 45 minutes to pass, and developer needs to run 10 builds a day on average, just as part of their work. If you can cut that time in half, you could be almost doubling the capacity of your organization in terms of the productivity of the developers and what they’re able to output.

So that’s a simplified example, but there’s many opportunities like this for, I think, most companies if you go looking. I think that’s where you should begin is identify as a single project, and pitch that to leadership. Pitch that to business stakeholders. Use that win to then start building that muscle, start building that awareness and start building that credibility and buy-in for the power and impact that can be achieved when you invest in developer experience. I think that’s where I would sort of begin in terms of baby steps.

Henry Suryawirawan: So I think that speaks to a lot of leaders out there, right? So don’t chase the buzzwords. Start from the real problems that you see in your engineering teams. You mentioned things like people are leaving, so that’s probably a warning sign already, although it’s like probably too late by then. Or slow builds. Tests are flaky. Those kinds of things. Focus on solving that because that will definitely give a lot of impact. If people are complaining a lot, when it gets resolved, I’m sure it will give a lot of impact and satisfaction. But the other thing that I learned from my journey as well, these things tend to change over the time. So maybe this day, it’s about built. Tomorrow, it’s documentation. The next could be test. When you write a lot of test, maybe they’re flaky or take too long to pass. I guess, all these things are like evolution, right? You probably never get a perfect state. I’m sure maybe some organizations think, oh, they want to have a perfect developer experience. I think it’s kind of like difficult. Especially these are people problem as well. Socio technical problem. So I guess these things tend to change.

[00:40:49] Developer Experience Team

Henry Suryawirawan: And you mentioned in the beginning, some companies actually invest in building a dedicated team. Maybe they are called like enablement team, developer experience team. So for companies who are at that stage, maybe they are big enough or they have hundreds of engineers, when do they need to start investing building such team? And how should they go about it?

Abi Noda: Well, most leaders who have developer experience teams will typically talk about how they should have done it sooner. So I think there’s no sort of single track answer to that question. But I think as soon as you’re probably hitting around 50 to 80 engineers and growing, and also probably hit a point of like product market fit in your business where you’re not just wildly scrambling to ship things that you might throw away. As soon as you sort of have a longer term view on the business and plan for growth, I think that’s the right time to take a good hard look at how can we get double the output out of our organization.

That’s what developer experience is about. It’s about taking the people you have and empowering them to be able to do so much more. It’s about getting the most out of your investment if you want to think out of business terms. And so, I think the right time, as I said, if I sort of take the average of all the different companies I’ve spoken to, I think around 50 to 70 engineers post product market fit is a good time to start establishing a muscle around developer experience. I wouldn’t necessarily say you have to form a team because there’re many different ways to tackle developer experience. It doesn’t have to be a dedicated team, necessarily. But I think at that point you would definitely want a senior leader really taking a good hard look at the developer experience and looking for opportunities. Starting to build roadmaps. Starting to think about strategy. And considering forming a team around it, if that’s the right way to tackle the problem.

Henry Suryawirawan: So yeah, I think I also myself face this challenge with my engineering team. Sometimes, yeah, they’re just anecdotal complaints, okay, we had bad process here, bad process there. But where to tackle? Sometimes it’s hard and there’s no dedicated team as well. Every engineer is just churning out features over features over features. Who will build the enablement team? So I guess this is something to think about for those leaders, right? When is the right time for you to start really investing? Maybe create a more dedicated team, build more purposeful tools, so that engineers are more productive.

[00:43:08] Building a Case for Developer Experience

Henry Suryawirawan: Maybe this is a little bit more of a challenging question for you to answer, but what if a company doesn’t want to invest in developer experience? They think, oh, they’re just engineers. They’re just here to produce code. So what would be your advice or maybe tips here?

Abi Noda: Oh, that’s such a good question. I mean, I think of this one company that I know. I think they’re around 300 engineers. There’s a couple developers there trying to get a DevX function off the ground and doesn’t sound like leadership super bought in. And I really feel for those developers. Clearly, there’s huge opportunities to improve the productivity of the organization, but executives are skeptical. To be honest, I don’t know if that’s always a problem you can solve from the inside. You think back 10 years ago, even DevOps was hardly a term. So I think in a sense, shamelessly, I would say, calling up a company like us to come in and talk to your executives and get them bought in to the idea of developer experience. That’s part of what we do, really, is help developer experience teams get buy-in from executives by taking a very sort of precise and methodical approach to developer experience that’s really aligned with the business.

I think it’s just hard sometimes as developers, we’re not used to some of the corporate politics and the ways of the businesses kind of need to operate and determine where to make investments. So that’s where I think developer experience has trouble getting off the ground. Sometimes it’s really speaking in terms that the business can understand. Because I think if you use the right language and present developer experience in the right way, the ROI is very obvious and the organizations that do invest in developer experience know that. And so, I think it’ll take a little bit of time before developer experience sort of goes fully mainstream, if you will. You know, I talk to people all the time about this, and I think most in the industry would agree, like developer experience is still kind of early on that adoption curve. And so, if you’re out there listening to this and you’re having trouble getting buy-in from your executives, that’s to be expected. Not every company, not every leader is going to get it. But try to make the case in terms that the business can understand. Not using buzzwords. Not pointing to trends or fads in the industry. But just point out the dollars that are being lost or the dollars that could be gained back if that latent sort of potential could be unlocked within the organization through improvements to developer experience. That would be my advice.

Henry Suryawirawan: So I know it’s a hard problem. Thanks for trying to answer that. So I think any kind of organizations when they build tech teams, they always think about feature teams, right? We want a feature teams, product teams, whatever that is. I’m sure one day it will reach a point where not just feature teams that they will build, they will also build this developer experience team or engineering enablement team straight away from the get go. And I guess I would also rely on researchers, your company and some of the researchers behind the company, to actually publish this kind of paper so that it becomes revolutionary, so people take it just like DORA metrics, right? Oh, we have these four key metrics that we also should think about when we talk about developer experience. So I think the work that you do with your team is exciting. I hope that something good come out in the industry to improve on this.

[00:46:18] 3 Tech Lead Wisdom

Henry Suryawirawan: So Abi, it’s been a pleasure talking to you. I learned a lot about improving developer experience. So as we’re going to wrap up this conversation, I have one last question which I always ask for my guests, which is to share this thing called three technical leadership wisdom. You can think of it just like advice for listeners here to learn from your journey, your experience, or your expertise. So what will be your three technical leadership wisdom?

Abi Noda: Yeah. Great question. I think, on the topic we’ve been sort of discussing today, I would say, be very discriminant about how you choose metrics. There’s a lot of stuff out there. There’re a lot of vendors out there, including us. I’ll admit. There’s a lot of faux research out there. There’s a lot of good research out there. Know exactly what it is you’re trying to drive in your organization before you start picking up metrics. I see so many leaders shop for metrics instead of really think critically about the types of outcomes they’re hoping they achieve in their organization, and then ask if there are measurements that could help them achieve that. So that would be the first.

The second would be, surveys are very underestimated in our industry. I think it’s cause we’re programmers and we love streaming real-time data coming out of systems into pipelines, into D3 charts. But one of the biggest mistakes I’ve seen, and Nicole Forsgren would agree with this, we’ve talked about this is, we’ve seen corporations spend tens of millions of dollars trying to instrument their systems to measure the DORA metrics. When in fact, what they don’t realize is that even in her research, Nicole used surveys to measure the DORA metrics. And in fact, the whole last third half of the book goes into why and how she did that. And so, surveys are a way to measure both subjective and objective metrics. There are limitations to using surveys. For example, you can’t measure things on a real time basis. You can’t run surveys every hour. You also can’t measure things down to the millisecond with surveys. But there’s a lot you can measure, and I think a lot of leaders are spending too much money, too much hassle trying to get poor metrics that won’t even help them. When a more obvious starting path, if you think about it, is to just start asking your developers questions about what’s getting in their way? How their processes are? How their tools are working for them?

The last sort of word of wisdom I would share here would be definitely to just follow the research that’s coming out in the industry. Whether it’s research on the different types of agile, Scrum, kanban, practices, or whether it’s research on algorithms and hardware systems. There’s so much amazing research coming out of academic institutions as well as companies like Microsoft and Google. That’s one of the things I’ve been working on is I have a newsletter where we write summaries of research. These research papers are pretty dense to read. It’s not really casual reading, and so, we’re trying to make it a little more accessible for practitioners. I mean, we have a multi-year long backlog of papers we’re looking forward to sharing, but there’s really amazing stuff. As we talked about earlier, it’s so important because the research often debunk some of the sort of trendy opinions of the year as far as sort of what you’re hearing on blogs, that you’re reading on Twitter. So I think more leaders, more practitioners, even developers should be following the research that’s happening and I think would be able to use that within their organizations to make the case for things like developer experience much easier if they were leaning on research rather than, for example, tweets.

Henry Suryawirawan: I really love all your wisdom. It’s really unique. So some of key takeaways for me is like, first, don’t shop metrics. Think of what actually the underlying problems you’re trying to solve and try to find measurement out of that. And survey is underestimated. Yeah, I agree with you. Sometimes we engineers, especially if you have a dedicated team, they will tend to build technologies to solve a problem, and they will take a lot of effort and time to actually build it. But, at the end of the day, maybe survey itself suffices, right? So the ratio of input and output probably is worth it, so just do survey.

Lastly, I’m a subscriber to your Substack newsletter. I find it really interesting. You cover a lot of software engineering kind of related research. I wasn’t aware of those kinds of research before, and luckily, you do all this so that I can read it in an easier way. Not like in a dense, complicated way of explaining things. So thanks for sharing that, and I will put a plug for the newsletter so people can also subscribe and read from there. I think it’s really useful for engineering leaders.

So if people want to connect with you or talk more about developer experience online, is there a place where they can reach out?

Abi Noda: Yeah. Always love connecting with people. So I’m on Twitter, AbiNoda. You can email me abinoda@getdx.com or on LinkedIn. Yeah, always happy to connect with people. Share advice if I can help and hear what people are up to in this space.

Henry Suryawirawan: And Abi also has a podcast. So if you are into developer experience, do check out Abi’s podcast as well.

So thanks again for your time, Abi. I’m really excited about all the research that your team are doing. So good luck with all that and looking forward to the paper being published next year.

Abi Noda: Thanks so much. Thanks for having me. This is fun.

– End –