#134 - A Developer-Centric Approach to Measuring and Improving Productivity - Margaret-Anne Storey & Abi Noda

[00:00:58] Episode Introduction

Henry Suryawirawan: Hello again to all of you, my listeners. You’re listening to Tech Lead Journal podcast, the podcast where you can learn about technical leadership and excellence from my conversations with great thought leaders in the tech industry. If you haven’t, please subscribe and follow the show on your podcast app and social media on LinkedIn, Twitter, and Instagram. And to appreciate and support my work, subscribe as a patron at techleadjournal.dev/patron, or buy me a coffee at techleadjournal.dev/tip.

My guests for today’s episode are Dr. Margaret-Anne Storey (aka Peggy) and Abi Noda. They are the co-authors of the recently published ACM paper “DevEx: What Actually Drives Productivity”. In this episode, we discussed how we can better measure and improve developer productivity using a developer centric approach. Peggy and Abi first began by explaining the importance of socio-technical factors in software development. They also shared their view on the well-known SPACE and DORA metrics and pointed out the danger of misusing and abusing the DORA metrics. Peggy and Abi then explained the three core dimensions of developer experience from their latest paper, which are feedback loops, cognitive load, and flow state. Towards the end, they also shared tips on how we can start measuring developer experience, including how to conduct developer surveys properly.

This is such an amazing discussion with Dr. Margaret and Abi, and it is also my pleasure to have Abi for the second time in the podcast to continue our discussion about developer experience. I hope you enjoy listening to this episode and learning a lot from it. And if you do, please share this with your colleagues, your friends, and communities, and also leave a five-star rating and review on Apple Podcasts and Spotify. Let’s go to my conversation with Peggy and Abi, after hearing a few words from our sponsors.

[00:03:11] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I’m really excited. We have a repeat guest and also a new guest at the same time. So we have two guests in the show today. So the first one is Abi Noda. If you still remember, we had him in episode 112. It is actually one of the fastest growing episodes. So now, it’s already, I think, in top five, top six episodes. And it’s really great to see, because that means the topic developer experience is actually really trending these days.

And I also have Dr. Margaret-Anne Storey, or some people call her Peggy. So Dr. Margaret is actually one of the authors of SPACE paper as well. So if you still remember, we have DORA and SPACE, right, in the industry about developer experience and developer productivity. So we will be talking a bit about SPACE, but both of them come here actually to announce a new paper that they have just published recently. Peggy, Abi, thank you so much for this opportunity. Looking forward for our conversation.

Abi Noda: Yeah, thanks for having us.

Margaret-Anne Storey: Yeah, it’s great to be here. Nice to meet you, Henry.

[00:04:09] Career Journey

Henry Suryawirawan: Since I already had Abi before, so maybe Peggy, this time I will ask you a little bit about your career journey. Maybe your background, so what led you to SPACE or things like that. Maybe you can share with us a little bit.

Margaret-Anne Storey: Sure, no problem. So I’ve been actually a professor for about a couple of decades now. And I’ve been a professor at the University of Victoria, studying tools that developers use, trying to understand what makes them more productive, what makes them happier in the work that they do. And in my early years, I studied a lot and designed tools to help developers be more effective and how to collaborate with each other. Then in the last few years, I’ve turned to studying more and understanding about developer productivity, developer satisfaction, and more recently developer experience.

So during my journey, although I’ve been an academic, I’ve worked a lot with industry. So I spent many years working with IBM and the early days of the Eclipse IDE. And then in recent years, I’ve been collaborating a lot with the engineering tools team at Microsoft, consulting with them quite a lot and looking at the different tools that they use and studying developer productivity. And I’ve also worked with some smaller companies and notably with Abi on getdx, helping to do some research there behind developer experience, which I’m super excited about. And so happy to be here today to talk a little bit about that paper and also about the SPACE paper and our new paper that’s coming out.

Henry Suryawirawan: So many papers you have produced, I guess. So maybe a little bit, how did you meet Abi? How did you end up collaborating with each other?

Margaret-Anne Storey: Well, actually, I had heard about Abi from Nicole Forsgren before I met him. And then Michaela Greiler, who’s the first author on the developer experience paper. She and I worked so many years before on developer testing research. And Michaela reached out to me to say we’re doing this study with developers and interview study to understand developer experience and invited me to join their collaboration. So through that, I met Abi and we still continue to work together around getdx and brainstorming around the theories and the survey questions and so on.

[00:06:19] First Developer Experience Paper

Henry Suryawirawan: So you mentioned a couple of papers, right? So before we go to the new paper, maybe a little bit of flashbacks what you have collaborated in the beginning, the first paper. Maybe tell us more, a little bit about that paper. What did you find as an insights and maybe later on, why did you come up with a new paper? So maybe Abi or Peggy here, you can chip in here.

Abi Noda: The initial paper Peggy and I worked on was really focused on understanding what developer experience was and defining it. And so in our investigation we first looked at existing literature to build a working definition of what developer experience is, and then our investigation focused on understanding what actually affects developer experience.

And we know so many both human and technical factors affect developer experience. But the real question was, which are the most important or relevant factors that affect developer experience? And so through that investigation, the key findings of our paper were a set of socio and technical factors, which affect developer experience as well as an understanding of why developer experience is important for organizations and why so many organizations struggle to identify problems with their developer experience and make improvements.

[00:07:37] Socio-Technical Factors

Henry Suryawirawan: So you mentioned about socio-technical factors. Maybe some people have heard about it, maybe some don’t. Why socio-technical factors and what is socio-technical factors?

Abi Noda: The point around socio-technical factors is really important, because one of the biggest misconceptions and myths around developer experience, I would call it really an anti-pattern, is that developer experience is just about tools. You see so many organizations out there trying to improve developer experience, and when they talk about developer experience, they’re veering it through this narrow lens of developer tooling.

But in fact, in our research and the decades of research Peggy has worked on prior to this, we’ve found that human factors, things like collaboration or psychological safety or just the exchange of information between teams and people have a really important effect on developer experience. And I would personally argue have an even greater effect on developer experience than the typical tooling problems that organizations focus on.

Margaret-Anne Storey: Yeah. There’s a lot of the teams and I guess even managers and leaders that I’ve worked with, sometimes I go into a company and they say we wanna improve developer productivity or we wanna make our developers happier, and so we’re looking at improving their tools.

And I’ll often say it might not just be about their tools. There might be social factors. And they’re like, oh, don’t tell me about those fuzzy, human and cultural things. I don’t wanna hear about that. And I’m like well, I’m sorry that we’re probably gonna end up finding that some of those things matter.

But there’s a great expression that I read about the other day. And this is that there is no technical without social and there is no social without technical. All of these factors even that we bring up, you can’t really say this one is just technical and this one is social. They’re all somehow socio-technical. They have both aspects.

Abi Noda: I loved that. And I was just gonna add, I also recently thought of an example of this, which is code quality. We think of code quality as a technical problem, but in fact, the real challenges of code quality are around humans understanding other human’s code and working on other human’s code. And so is code quality really a technical problem or is it a human problem? And that’s an interesting idea I’ve been thinking about.

Margaret-Anne Storey: That’s a great example! And Abi, I know you know about the research that we did when we looked at how developers and managers define not just productivity, but also quality. And a lot of the developers and managers mentioned in their definitions of quality that it’s about writing code so that other people can build on top of my code. Or so that other people or even I can understand it in the future and evolve it.

So I thought that was super interesting. It wasn’t just about, how secure it is or how robust it is, but more this kind of thinking about the bigger group of people that have to understand it.

Henry Suryawirawan: Right. So I think recent papers, books, also mentioned about this socio-technical aspects. So team topologies is one as well, right? So whenever you work with multiple people, not just one, right? If you’re a solo developer, you build hobby project, maybe arguably you don’t need this social technical aspects. But once you work with someone, right? I think these social technical factors become more important. So yeah, team topologies mentioned about that and maybe there are a few other research including this paper that you just published.

[00:10:54] Examples of Socio-Technical Factors

Henry Suryawirawan: So maybe if you can give us some examples of these socio-technical factors that you identified in your first paper. I know there are 25 of them. You don’t need to list all of them, but you can give some, maybe, your favorites.

Margaret-Anne Storey: Some, I think, most of them, when you read the factors, so I think, yeah, I think we had about 25 factors that come up. Few of them were really what you would think of as technical. So you know, things like codebase health, the development environment, and frictional releases and so on are more on the technical side. But a lot of the factors that came out were things like, for developers, having clear goals and knowing the scope of the work that they had to do and understanding the requirements. Having reasonable deadlines being set for them by their team and by their managers. Having a say on the roadmap and feeling like they played a role in what was going on. And then others like having autonomy. Having a say in what was gonna happen. That aspect of psychological safety, that I know Henry you might chat about a bit later on as well. And things like feeling connected to other people in their team. Feeling heard and, getting help and giving help.

So a lot of the factors are more, if you like, on the social side, but there are of course that technical aspect to them as well.

Abi Noda: Kind of piggybacking off of what Peggy talked about, it reminds me of an experience I had at GitHub where leadership wanted to improve the DORA metrics. And I know we’ll talk about the DORA metrics later. But in particular, I was actually put in charge of trying to understand how to improve our lead time. And at the time, the business was looking at lead time as a technical problem. A problem with our development tooling and CI/CD processes, and there was a lot of opportunity to improve that.

However, when I started working on the problem and talking to managers and leaders across the organization and asking them, what do you think would improve your lead time? The most common responses were actually human factors such as scope creep and poor product management. I remember the line from a leader saying, our biggest problem is that tasks just take forever, because we keep changing the scope as developers work on them.

And so again, this ties back to the earlier point around developer productivity and developer experience are about much more than just tools. And organizations that just look at it as a tooling problem are missing out on a lot of opportunities for improvement.

Henry Suryawirawan: Thank you for that perfect example. I know that we have covered before, right? What are the bad metrics that we shouldn’t measure. So thanks for giving some good examples in this episode, right? So for people who are still thinking that it’s all about tooling, it’s all about software metrics. Please don’t get it wrong. Try to also include the socio-technical aspects.

[00:13:35] SPACE & DORA Metrics

Henry Suryawirawan: So you have mentioned DORA, and we have also the SPACE author here. So these are the two, maybe, most important papers that people quoted a lot. I know DORA is quoted even more than SPACE, so let’s start with SPACE first. Why isn’t it picking up in the industry, like people talking about SPACE metrics. So maybe Peggy, you have some opinions here.

Margaret-Anne Storey: The reason we came up with SPACE really was to encourage the industry and to encourage managers and leaders to not just think about productivity according to things that you could easily measure, such as lines of code or pull requests or code reviews done or features shipped. And to really broaden what productivity means. So we, all of us, there were many authors on this paper and together we brought a lot of experience from studying developer productivity and satisfaction together. And we had seen, that this word “productivity”, and perhaps Abi talked about this before when he visited you, I’m sure that he did, that this concept of productivity, what does it mean? It’s hard to define. It’s an elusive term and what we’ve found is that it means many different things. And that was why we came up with the SPACE framework was to show that when you’re thinking about being productive, there are really these different dimensions that you need to think about.

So we’ve already been talking about one, right? This importance of collaboration. So it isn’t just writing lines of code or shipping features, but it is thinking about how you help others get unstuck in their work. Or how you help others onboard to your code or the effort you go to write code so that others can understand it and leverage that code in their own work. Or rushing to do something so that you unblock somebody else.

But there’s also other aspects to productivity. How satisfied you are. So some earlier research that I did and research that others did as well, saw that happier or more satisfied developers also feel more productive and vice versa. So feeling satisfied and feeling happy and feeling joy in your work is important in terms of being productive. And as well those feelings of being in flow and not being interrupted and really having that joyful experience of writing code and making progress and having impact in your work. Those are all important aspects of productivity.

Now, in the paper we do talk about some metrics. And I have come across quite a few companies that are using SPACE for defining metrics, and I’m always a little bit worried when I hear that because, on the one hand we think that if you are gonna define metrics, at least use, and we came up with a number, at least use three dimensions, right.

Be more holistic. Don’t just focus on activity or outcomes, such as quality of feature shipped. But look at these other aspects of developer satisfaction, collaboration, and also efficiency and flow. And it’s good to see that. It’s good to see that some organizations are coming up with metrics and using multiple dimensions across SPACE. I’ve talked to quite a few organizations that are doing so. But it wasn’t the main reason that we didn’t try to say “Here are the metrics. Here’s how you should define it.” It’s more about if you are thinking about productivity and making a change, then you should think about these different dimensions.

So for example, if you’re, say, introducing a new code review tool, and you wanna evaluate, or understand the impact that tool has on developer productivity, then think about it not just in terms of lines of code written or number of bugs when you ship the code. But think about it holistically. What is it meaning? Maybe you are improving the speed of something, but maybe developers are not as happy, right? Or maybe they’re not seeing the impact of their work or something like that. So that was the reason was to try to steer people towards looking at those different things.

So I think DORA is quite a bit more specific in what it looks at, whereas SPACE is very broad, very holistic.

Abi Noda: Henry, it’s an interesting question that you asked because I wouldn’t necessarily agree that DORA has taken off more than SPACE. I do think DORA has had many years ahead of SPACE to proliferate. But to what Peggy was saying, I think one of my personal observations is that SPACE is complex, right?

That the point of the SPACE framework is to show people that developer productivity is complex and multidimensional. And I don’t think that’s the answer people always want. In contrast, the DORA metrics are easy. There’s four metrics and they’re “scientifically proven”. And I’ve heard people skeptically refer to them as the easy button for executives to try to measure engineering performance. Of course, we’ll talk later about why that’s actually difficult to do.

But I think SPACE looks at developer productivity holistically. Whereas, like Peggy said, the DORA metrics are out of the box metrics that you can use. And I think that’s part of the reason why you see many organizations using the DORA metrics and trying to figure out developer productivity still.

Henry Suryawirawan: So actually I have similar thought, or maybe my assumption, right? So coming back to our earlier discussion, I think it’s about people in the industry still not thinking software development as a socio-technical aspects. And also thinking about, what’s the easiest metrics that actually I can quantify in the organization? Not really looking into people’s satisfaction or these kind of things. Sometimes it’s really complex, like you mentioned multi-dimensional. So that’s my assumption anyway.

[00:18:52] Misuse and Abuse of DORA Metrics

Henry Suryawirawan: But coming back to DORA, right, I know that you have maybe a couple of episodes in your own podcast talking about this, right? How DORA is being misused in some organizations. Maybe you can give some of your summary or thoughts here. What are some of the examples why people ended up abusing or maybe using the DORA metrics wrongly.

Abi Noda: Yeah, I’ve been able to have really great conversations recently, including with Nathen Harvey, who currently leads DORA at Google and has really interesting perspectives on the misuse and abuse of DORA metrics. But before going into that, I just wanna say that, we both, Peggy and I work with Nicole. Nicole is amazing, and DORA was and is still a groundbreaking research. So this isn’t meant as a critique of DORA, but rather that the DORA metrics, or which were known as the four key metrics, have gotten taken a little bit out of context and gotten a little out of control. And what we see today is that many leaders are struggling to actually gain value out of them.

I’ll maybe just begin with my personal story. I alluded to this at GitHub. At GitHub, we were going along this journey of transforming the organization, accelerating developer velocity. And like many organizations, our executives said we should measure the DORA metrics. And of course, this was well intentioned. At the time, the DORA metrics were seen and still are seen as the sort of gold standard of metrics. They’re easy, there’s four of them. You don’t have to think very much to put ‘em in place. And my role at GitHub at the time was to actually move those metrics and figure out how to improve developer velocity at GitHub. Long story short is that I ran into a lot of problems with how to do that with the DORA metrics. And the big problem I think is that what’s gotten missed about the DORA metrics is that they’re outcomes and not capabilities. They’re not really metrics that you should be setting as targets or trying to optimize in of themselves.

And people often talk about the book Accelerate. And they joke that executives only read page 19 and then put down the book. Page 19 is where the four metrics are described. They don’t talk about the broad swath of capabilities that the book talks about, which is what organizations actually need to be focusing on.

And at GitHub, the problem was once we had the DORA metrics, the next question was, so what? So how do we actually improve these? And the problem was that the DORA metrics didn’t really capture the context of what was happening across the organization or root cause. And thus what I was left doing is really going around the company interviewing developers and teams to understand, okay, what is actually slowing you down?

And that’s part of what led me to work with Peggy and Nicole now on focusing on that exercise. Me going around and talking to people, which we’re now essentially calling developer experience.

Henry Suryawirawan: So I really love that joke. When I heard about that in your episode with Nathen as well. So people only go to page 19, see the table, right. You have high performers, medium performers, low performers, and you have some kind of a metrics that you can quantify, where do you categorize. So I guess for leaders, they will just see, okay, where is my team now and we should improve to the high performers. So I think, yeah, many people don’t read fully in terms of paper.

And I’ve read the recent State of DevOps Report 2022. I think there are a lot of insights as well, and especially every year they come up with a different theme, a different analysis. I think those kind of analysis is also interesting. And there are many learnings that we can do outside of just the four key metrics.

[00:22:29] New Developer Experience Paper

Henry Suryawirawan: So let’s dive into the new paper, right? I think the paper title, I know that probably it will change, so it’s more about developer experience, what actually drives productivity and how to measure it. So maybe a little bit of intro in the beginning. What is this paper about? What is your findings or insights? So maybe, yeah, either one of you can chip in here.

Abi Noda: The reason we decided to write this paper was to really build on the work we did in our initial research paper. In that initial research paper, we defined what developer experience is and what it’s really made up of and what it looks like in organizations and why it’s such a challenge for them to actually improve it. With this paper, we wanted to create a resource that was more practical. Something that could easily be applied by leaders in organizations. And this paper really encapsulates both the research findings from our initial paper as well as the experiences we’ve now had working with many organizations and partnering with them to help them understand and improve developer experience.

And so this paper builds on our experiences and research to hopefully provide a framework or set of frameworks that organizations can go ahead and apply to start measuring developer experience and also start improving developer experience within their organizations.

Margaret-Anne Storey: Yeah, to me, I think the key behind this paper. The first paper was really about understanding the factors that impacted developer experience and also teased out a lot of contextual characteristics. How senior a developer is and what kind of size team they’re working in and their level of management. All of these things that affect how different factors may play a role in their experience.

And we also, from that work, identified barriers and coping strategies that developers would use. And the original framework was intended to be quite actionable, because it did identify barriers and identified these strategies that developers would use, but it’s also hard to put it into play.

And so I think this new paper, which kind of distills three dimensions of developer experience into feedback, cognitive load, and cognitive flow. These three dimensions are easier to wrap your head around and to identify how to measure them. So that you can make changes and hopefully improve developer experience by focusing on those three things.

[00:24:55] Developer Experience

Henry Suryawirawan: So I know that some people probably listen to this episode, like coming from a new perspective, right? They may not understand what is developer experience. I forget that we have mentioned developer experience so many times, but some people may not be familiar with it. Maybe if you can do a quick refresher, what is actually developer experience and is there anything changing after this new paper come out?

Margaret-Anne Storey: The high level definition of developer experience that we use is, it’s a bit similar to the user experience concept that you get from human computer interaction or UX research, and it’s all about how a developer feels about, thinks about, and values their work.

So it really does look at the cognitive perspective of the developer and these different sort of psychological aspects. And together, those aren’t distinct, but together those insights build a picture of the kind of experience that a developer has. And in particular, the values dimension is really important.

And I mentioned this earlier, that our earlier work showed that, developers when they feel that their work has impact, they feel more satisfied, but they also feel more productive. And this is really important in terms of motivating them to do their best work. So developer experience really focuses on how the developer feels and thinks about and values the work. I know I’m repeating myself, there really are these three pieces to it.

And then this new paper kind of focuses on those three aspects and builds on them to see what are the things that we can measure? Because those are fuzzy, right? How do you value your work? How do you think about your work? It’s hard to turn those into things that you could measure or even reason about how you could make improvements. So the three dimensions in this new paper is about really thinking about, okay, what is it we can really measure, understand, and then suggest changes for.

Abi Noda: Yeah. I loved Peggy’s overview of the scientific perspective on developer experience. I can maybe speak a little bit to the layman’s view of developer experience. Developer experience is a buzzword now, and I think it’s used in a couple different contexts.

So for one, to build on what Peggy said, developer experience is about the lived experiences of developers. And Peggy provided a sort of rooted in psychology, but a way of thinking about that, which we build on in this recent paper. Developer experience is also, I think, a practice though in the industry, it’s an approach. And in the paper we refer to it as a developer-centric approach to understanding and improving productivity. And you’re seeing that play out in many different ways. There are now teams out there called DevEx teams. Gartner recently ran a survey and found that over 90% of organizations are either have planned or are establishing DevEx initiatives.

And this goes beyond even what we’re talking about today in terms of what developer experience is. Developer experience is something that is happening. Organizations are investing in and planning initiatives across the industry.

Henry Suryawirawan: Right. And the overarching goal why people start to invest a lot in developer experience, because I think it has been proven that productive developers definitely bring a lot of values to the company and in fact ended up with increased business performance, either profit or whatever it is that you’re doing as an organization.

[00:28:11] 3 Core Dimensions

Henry Suryawirawan: Peggy, you mentioned three core dimensions. Let’s just go dive into those three. So you mentioned about feedback loops, cognitive load, and also flow state. So maybe, yeah, can you explain maybe one by one what is feedback loops and why this important for us to think about measuring it.

Margaret-Anne Storey: Yeah. So developers, how they work and the work that they do, they rely so much on feedback. Whether it’s getting feedback from the system or whether it’s getting feedback from other developers. This is gonna impact. Anytime you make a change in the code or you work on a feature, you need to get feedback that you’re doing the right thing, right? That this is heading in the right direction. Whether I’m waiting for somebody to review my code or waiting for tests to run or waiting to get feedback from end users. This is an important part of giving me that feedback that I’m on the right track, right?

And so these, the speed of these feedback loops, and this is what agile development was all about, was increasing that speed, right? So that you would get that feedback quickly and you would know that I’m understanding this correctly and getting the right value, seeing the right value from the work that I’m doing, and I’m meeting other needs. And of course, CI and CD is all about that as well. Speeding up the feedback that you get.

So the feedback loops was the first kind of dimension that kind of emerged from using the DevEx framework in the tooling that Abi and his colleagues created.

And the second one is cognitive load. Sometimes this is also referred to cognitive fatigue. So this is, the amount of mental processing that is required for a developer to perform a task. And of course, software development today is a very challenging cognitive task to do. You have to understand a lot of things. A lot of your work is understanding not just the code that you’re working on, but also all of the code that your piece of software depends on. And there can be many different layers of abstractions, not to mention all of the tooling that you’re using and all of the other things that people in your team are working on. Some cognitive load is good. We feel happy when we have some pressure on us. But if there’s too much, then you feel overwhelmed or you get confused and you forget. So this is the second one that’s really, really important.

And then the third one is this notion of cognitive flow which I’ve been looking at for actually over 20 years now. This idea of supporting developers getting in flow. And often people think, oh, that just refers to not being interrupted. But being in flow is a lot more than that. That feeling of being in control of what you’re working on something at the right level of difficulty, feeling like you’re making progress, feeling like you’re creating something that’s worthwhile. That kind of captures that whole kind of flow notion. And that can be supported, by many things such as having autonomy over your work or having clear goals, as well as having that uninterrupted time to do that.

So together, these three things, they’re not independent. They are dependent, right? If you make a change that impacts one of them, that’s gonna impact the other two. So together, we feel these are the three most important kind of ways that we can make improvements to developer experience.

Abi Noda: One thing I would add to what Peggy described is going back to an earlier theme of both social and technical factors spanning across these three different dimensions. For example, if you think of feedback loops, again, most people think of feedback loops in terms of the speed of tools or the speed of processes. But in fact, a lot of those tools and processes involve humanous aspects as well.

For example, in our paper we provide examples of feedback loops and tools, but we also talk about the feedback loops which exist amongst people. One obvious example would be waiting on peers for feedback on code for code reviews. But those types of feedback loops exist outside of code review as well. There’s usually feedback loops involved in QA, feedback loops involved in getting feedback from your actual users and customers or stakeholders. So that’s something I wanted to call out.

And the same applies for cognitive load and flow state as well. Again, with cognitive load, we like to think of developers scattered across too many tools and systems. But in fact, a lot of cognitive load has to do with communication between people or communication between people via code that is difficult to work in.

And as Peggy talked about at the end, flow state is the same thing. It’s not just around the sort of mechanical aspect of people being interrupted. It’s also about them feeling stimulated and fulfilled and empowered in their roles. Those are all key components that promote a state of flow state in developers.

[00:32:44] Optimizing Feedback Loops

Henry Suryawirawan: Thanks for giving a high level summary and what are the three dimensions. Maybe if I can just go back a little bit on feedback loop. So there are so many methodologies already exist. Like you mentioned about Agile, CI/CD, maybe Lean methodology as well that tries to optimize this feedback loop.

So I guess my question will be even though those methodology has been around for years, people are still struggling about feedback loops. One particular thing is like, how can an organization identify these kind of like warning signals that actually their feedback loop is not that optimized?

Another thing I would say, the DORA metrics also have these two key metrics that ties back to feedback loop, right? Which are the deployment frequency and also lead time for changes to production, right? So maybe yeah, you can give us some, I don’t know, like warning signals, “Hey organization, you are not optimizing this feedback loop.”

Abi Noda: Well, one of the themes of our paper is that the way to get signal about friction in the developer experience across all these dimensions is by asking developers themselves. And so my first recommendation would be if you’re trying to identify these constraints or areas of friction in the organization, ask your developers or interview them or shadow them. I know Peggy has done that sort of work at Microsoft before.

But moreover, you brought a really interesting point up, which is that DORA focuses on measuring some of these feedback loops. And that’s a great point. And in fact, within our example metrics that we provide for measuring things like feedback loops, we actually do provide some of the DORA metrics in the table.

And so earlier we talked about some of the challenges organizations have when they focus and optimizing things like the DORA metrics. But in fact, to me, the DORA metrics are really encapsulated in this bigger problem of developer experience. And they do a good job of measuring feedback loops, which is just one component that’s a part of the overall developer experience.

Margaret-Anne Storey: Yeah. And I think you said this really well Abi that it isn’t just about using DORA metrics. I mean they, they definitely, we need them right. They play an important role and they give us really important signals. But we need other kinds of signals. So even something like, maybe code review is happening really fast, maybe if you look at the system data and you say, oh look at how fast everybody’s reviewing my code.

But then you talk to developers and they say yeah, I get feedback quickly, but it’s not the kind of feedback I need. You wouldn’t necessarily be able to determine that unless you ask developers, is this feedback the kind of feedback you need to minimize delay and to help you move more quickly?

And it also isn’t just about speed. I just wanna put that in there. Always going faster isn’t always necessarily better. There’s this notion that we gotta keep speeding things up. But if you do that, developers may not stop to pause and think, am I developing the right thing? If they keep just rushing to deploy features and they don’t have time to stop and reflect and talk to people, then they may not be actually working on the right things that an organization or a team needs.

So it’s, again, it’s pretty complicated. And I think the point of this framework, and also SPACE as well, is to get people to stop and think about what are the important questions that we should be asking? Not just, let’s just pick some metrics, measure it, and we’re done. But really stopping and getting people to reflect and pause and say what do you mean? What do you mean if we do that, our quality will go down? What do you think quality means? So having those conversations.

Henry Suryawirawan: Thank you for emphasizing again, it’s not about speed, right? It’s not about you really shorten the feedback loops such that you can really deploy really fast. But it’s also about other aspects as well. I like to think that the DORA metrics is actually like a lagging indicators, right? There are so many other leading indicators that you can actually identify in the process that comes into the lagging indicators.

And I think another thing that I would say also about, some of these bottlenecks, right? Why your feedback look probably is not great, is probably also because there are dependencies, right? You work with multiple teams, you wait for some teams, the roadmap is not aligned. Or maybe you have too many siloed teams where you need to handoffs either code review, QA testing, security checks, and those kind of things. And also, another thing that I can think of from my experience is that too many work in progress. You struggle to juggle between multiple things, right?

So for people, if you wanna identify whether you have really great feedback loops, think about all these, delays, dependencies, handoffs, those kind of things. It’s not just about tools. How fast your CI/CD can process your new revision and then push it into production.

[00:37:06] Cognitive Load

Henry Suryawirawan: Let’s go to the cognitive load a little bit, right? Some people cannot wrap around what is this meaning about cognitive load? I know there are some research also talking about three different types of cognitive load: the intrinsic, the extraness, and germane. So maybe if you can tell us a little bit, what is a cognitive load actually?

Abi Noda: Yeah. I was getting really scared when you broke it down to the three different types of cognitive load. I thought you were gonna ask us for definitions. So I was pulling up my references here. But in the paper, we describe or define cognitive load as the amount of mental processing required for a developer to perform a task. And as you just alluded to, there are, in prior literature, different types of cognitive load and we briefly describe those in the paper.

So there is an aspect of cognitive load, which is just the inherent difficulty or complexity of a task or learning to understand an unfamiliar tool or framework. But one of the interesting aspects of cognitive load is that it also looks at, and when I say that I mean cognitive load theory, which really is the study of cognitive load itself. They do look at also how much processing it takes to actually intake or process external information. So for example, communication.

And so in writing this paper and diving further into cognitive load theory, it was really interesting to look at how these three different types of cognitive load all mapped back to common challenges in the developer experience.

Margaret-Anne Storey: And I think an important part of cognitive load to think about. Hans Dockter talked about this. I don’t know if you’ve seen his talks about cognitive fatigue. The fatigue that you experience when you struggle to understand something or you struggle to learn a new tool or to get something to work. That this really wears you down when you get stuck waiting for people or you don’t know why something works the way that it does. And you know that this can really, it can be exhausting for developers. And that, again, I’m gonna refer back, these three dimensions, aren’t independent.

If you’re working really hard to ship a product and you’re quickly shipping your features all the time, you never get that chance to relax and have that rest, right? Because you’re continuously deploying new features. And this was one of the feedback I heard from developers when I was at a workshop and I asked them, is it always better for you that we’re constantly deploying faster? And they said, we missed the old days when we used to ship our product three times a year, because we would have a party and we would relax for a few days. We could slow down and go, oh, we achieved that. And then they would be able to rest and then keep going again.

And the way that these things interplay is really important on developer experience. Things like also the code, technical debt and the code and the lack of documentation. These are also things that increase cognitive load.

Henry Suryawirawan: So about cognitive load, I think you mentioned about technical debt, maybe the complex code, right? And also unclear requirements sometimes, right? When you have a new, maybe business domains or business rules, but as a developer you don’t actually understand. So good documentation is definitely one of it. And I know these days people talk a lot about building platforms. Doing self-service, because there are a lot of technical complexities if you wanna deploy your code. For example, if you build data pipelines or data ingestions, maybe some people don’t actually have that kind of capability and they have to learn right? But having some kind of a self-service platform probably also helped to reduce this cognitive load. So thanks for sharing that.

[00:40:32] Flow State

Henry Suryawirawan: The last one, I think many people talk about it, even outside of the technical aspect, it’s about flow state. How can you actually be in a flow state? I know some people may refer to it a lot about meetings. So many meetings these days, right? So you cannot go into the flow. Maybe you can give us a little bit of tips here. How can we increase flow state for developers?

Margaret-Anne Storey: Yeah, definitely, structuring blocks of time, right? So that developers can have that. Be able to engage in that deep work is really important. But it’s not just as simple as doing that, right? And we talked about developers getting stuck waiting. They can’t move. They need help. They need somebody to explain something to them. Or maybe they need them to finish this change so that they’re unblocked and they can keep working on their piece. So there’s this tension between having these individual flow experiences and the team experiences.

And again, you know, I’ll come back to that, having these conversations about, having the terminology and saying, it’s great to have these big chunks of time. If you ask developers what makes them feel productive and what makes them feel like they’ve had a good day, it will be that they had a lot of time to code uninterrupted.

And so supporting that and recognizing that is important, that developers have these big chunks of time to work on their code is critical. But on the other hand, there’s a balance, right? Because you need the meetings between people to be able to make sure that the whole team has that experience of being able to move forward together. And that everybody understands everybody else.

Abi probably knows that I like to sail. And I like to race occasionally, although I don’t race so much these days. And one of the times that is the best when I’m on a boat is when you have this feeling of when action and awareness merge on the boat with other people that are sailing the boat. And nobody needs to say, oh, can you pull that line in or let that line out. Everybody just knows, right? That what you need to do, you’re aware of what’s going on, your situational awareness, and you know what’s happening.

Whereas in software development, unfortunately, you can’t see the entire course. You can’t see everybody’s hands doing the thing. You can get some insight into what they’re working on. But to get that situational awareness, you do need to have meetings. And so you do need people to come together and have those standups and say what they’re working on, and to have the ability to be able to say, “Can you help me with this? I’m stuck.”

Abi Noda: Yeah, as Peggy alluded to, there’s a lot of research, much of it led by Peggy herself on the impact of interruptions and things like meetings on flow. I think the elimination of impediments and interruptions is really important for promoting flow state with developers. But I also think one overlooked aspect is what I’ll just call motivation dimension of flow state as well.

It’s not just about eliminating the roadblocks for people to be able to be in that zone state. It’s also about giving them work in an environment where they are stimulated and energized and motivated to be in that state and be in the flow to do their work. And I think that’s something we touch on the paper. But I think going back to the earlier theme of engineering leaders really wanting to just focus on tools, I think it’s often missed that there is this very soft human aspect of flow that’s not as simple as just eliminating things on the calendar.

Margaret-Anne Storey: And if I can layer on top of that a little bit. I think that a really important part of flow is learning and supporting learning by developers. If you don’t have the time to learn how to use a tool effectively or to learn a new technology or to learn about a new API that you could be using, then you’re gonna be frustrated, right? You’re not gonna get in that experience of flow, cause you’re gonna be constantly misunderstanding what it is that you’re supposed to be doing. And getting stuck and having failed hypotheses about what’s gonna happen when you make this change and frustrated, right? And so I’ve noticed talking to developers at a lot of different organizations that the organizations don’t necessarily value time spent learning.

They’ll say that. I think big companies like Microsoft certainly do. They’ll have learning days and so on. But I think at a lot of smaller companies, it’s never recognized that, you know what, we need to just put the brakes on here, take some time to do some learning. We’re not gonna ship any code. Let’s do that. And then we’ll be able to get those flow experiences and increase feedback and reduce cognitive load later on if you take the time for learning. So learning is really important.

Henry Suryawirawan: Right. Thanks for the plug there. I always thought that one indicator when you can see a developer is in flow state is how many coffee cups they have or maybe how furious they type the keyboards with all the shortcuts. And how many swear words there are when they debug something. Jokes definitely.

Abi Noda: I would add, this might be insensitive, but it is funny, I mean, to your point about how many coffee mugs. Leaders always want, how do we get our developers to work harder? Lazy, they’re not working hard enough. But I actually think developers who are in a state of flow are the ones who are staying up really late at night and working, because they’re in this highly enjoyable, thrilling, stimulated, creative state. And again, I think flow state, that’s why it’s so important.

When people talk about how do we make our developers work faster? Or how do we get more out of our people, which is such a problem right now with the macro environment we’re in. You can’t just whip people into working and typing faster. But you can motivate them and stimulate them to be motivated in their work and therefore work harder and produce more for your organization.

Margaret-Anne Storey: And I think you said a really important word there, which is creative, the creativity. Often, the most creative insights that developers will have is when they’re in this flow state. So just to mention that it’s not just about output. Software development is such a creative activity. And so empowering and giving opportunity for your developers to get in that flow and creative state is really important.

[00:46:25] Importance of Culture

Henry Suryawirawan: Right. So Abi you mentioned about motivation, meaning, right? Peggy, you mentioned about creativity. So I have one question when I read the paper. One thing that popped up in my mind, where’s the cultural aspect? Things like psychological safety, because as a developer that might also relate to their productivity. If people are not safe in giving feedback or expressing concerns. Or maybe the leaders are the blaming type or they’re more like the top-down directive rather than talking with each other, collaborating. Where’s this cultural aspect? Or maybe it’s just immersed in these three dimensions, maybe a little bit on that.

Abi Noda: That’s a great question, Henry. Of course, in our previous paper on developer experience, we called out that psychological safety was perhaps the most important factor that sort of transcended all of the rest of the factors. And if you even look at the research behind things like DORA, they say the exact same thing. Culture always emerges as the most important factor and table stakes for creating effective teams and organizations. In this paper, you make a good point that we don’t explicitly call out culture as or psychological safety as a measure. And I think one of the reasons for that is that, I think A, that’s already a pretty common practice at most companies. Typically their HR organizations are already measuring aspects of psychological safety and culture, if you will.

But I’ll also provide a counterpoint to that, which is that we do focus a lot on measurements that capture things like satisfaction or the perceived ability to do high quality work and things like that. So I do think that we do capture culture across the three dimensions of developer experience. Although, we don’t maybe call out psychological safety specifically as a metric in our set of examples.

Margaret-Anne Storey: Yeah, I think you answered it really well. Culture is something that I care about a lot and having diverse teams and psychological safety. These are, they’re just a given. They are so important. And if you don’t have that, it doesn’t matter about anything else. Everything is going to break.

But even with cognitive flow, just to give some concrete examples of how kind of culture pervades everything to a certain extent. Having autonomy, right? Having that freedom to work on the same thing. Having that freedom to work on something and feeling that you can work on this thing, and it’s okay. If that’s not in place, then you’re not gonna have those experiences, right? You’re not gonna be able to say no to your team. I can’t help you right now. Or no, I don’t agree with what you’re doing. Everything else is just gonna fall apart.

But it’s a good question, Henry, because maybe we should talk about that in the beginning of the paper. That this is really important and we’re not just saying you can ignore that. You can’t. But we tried to find, I think these three dimensions are the things that are very specific to software development. So looking at the lens through software development, culture and collaboration and communication, those are important to every kind of work, right, or every relationship.

And so we were trying to highlight what are the kind of the actionable things that you might be able to measure and then change and improve in a software development context.

Abi Noda: Yeah. I think I maybe jumped the gun talking about the example metrics we provide in the framework. But even scrolling back through our paper as we’re discussing it, I do think we touch on culture throughout the paper. And maybe we don’t call it culture specifically when we refer to it, but when we talk about things like collaboration or autonomy or even flow state and motivation and providing stimulating work. I think those are all things that really relate back to what people would conventionally refer to as culture within an organization.

Henry Suryawirawan: Thanks for clarifying that. Definitely, the reason I asked is because some people don’t also get confused. Okay, these are the new metrics now, but forget about the cultural aspect. Even the DORA metrics, as you mentioned, right? It always mentions about culture. This Westrum organization culture, the three different types of the culture, right? So how organizations should also create a conducive culture for people to thrive, right? Not just again, focusing on the new metrics but also thinking about the people aspect.

[00:50:27] Measuring Developer Experience

Henry Suryawirawan: So I know we’ve discussed a lot of the theoretical aspects. Some people here are waiting how can we measure this developer experience? And in your paper, actually, you also advocate a practice. Maybe tell us how we can measure our developer experience.

Abi Noda: Yeah, this is the part of the paper I’m particularly excited about, because in the intro of the paper we talk about, again, how there’s been this elusive problem around how to measure and improve productivity. And we call it at the beginning that we hope that this paper does provide a fairly strong solution and answer to that eternal question or elusive question of what should we measure and focus on to improve productivity. So in the paper, what we provide is a set of example metrics across the three dimensions of developer experience we just talked about. And within those dimensions, we focus or suggest measuring two different aspects of each of those dimensions.

One aspect is what we call perceptions. And these are what some people would refer to as subjective metrics or perceptual or attitudinal metrics. These are things that can only be measured by asking people or conducting surveys. In addition, we also recommend example metrics which measure workflows. And these can be measured through either surveys or through systems or through just observation. But these are the actual sort of what we people would refer to as objective metrics or how much time something takes or how many steps it requires.

And it’s important to have both these perceptual and workflow measures when understanding developer experience, because the two really counterbalance each other. If you’re only measuring a developer’s sentiment or perceptions, you might miss out on opportunities that are manifesting themselves within their workflows. And on the other hand, if you are only focused on workflows, for example, DORA metrics, you potentially miss out on the bigger picture that you can only discover through actually asking developers about their experiences.

Margaret-Anne Storey: Yeah. And if I can add to that, the paper and the table of metrics. It’s a great start, but it’s still gonna require some work by somebody trying to understand some aspect of developer experience or trying to understand what happens when we make this change. I don’t think any organization would just take this table and take the examples that are here and measure them. At least I hope they don’t, right? Because they first have to decide what is it that we’re trying to achieve, right? Are we trying to understand or measure? And then maybe there’s some change, right? There’s some new tool or there’s some new practice that’s put in place. And then to decide which metrics should we use according to these six cells, right? Should we measure to see what is the impact of that?

And I think it’s really important to think along, as Abi said, the perceptual and the sort of the workflow or the more objective kind of measures. But to also think about the three feedback loops, cognitive load and cognitive flow. Because you could trick yourself into thinking I’m using this new tool and look at how much better everything is and look at how faster developers are writing code and shipping features to their customers. And meanwhile the developers are like, totally unhappy and they all leave.

So you have to take this holistic look and this is our attempt around doing that, right? But there’s still, it’s not a solution. Again, it’s complex. It’s not as easy as the four DORA metrics that you can take them and find a way to operationalize them. This is still gonna require some work, right. Abi, would you agree with that?

Abi Noda: Absolutely. Yeah. I’m glad you called that out. Because as we’ve seen with DORA, as we’ve seen with SPACE, and as we’ve seen with so many other things in tech, there’s this tendency to just blindly copy and paste solutions that other people have come up with. And this framework is, that’s why we call them examples, as you did in SPACE. And this is really, I think just a starting point to inform the way organizations think about how they should approach measurement of developer experience in order to achieve the outcomes that they’re hoping to drive.

[00:54:29] Conducting Survey

Henry Suryawirawan: I really love when you touched on about these two aspects, right? Perception, the subjectivity part, and also the workflows. A lot of people, especially engineers, we like to just build solutions and we focus a lot on workflows, right? Because we think we can think of something, measure from systems, data metrics, and then come up with a dashboard. But actually the perception part is also important. And don’t forget, it may be the easiest one that we can do, right, rather than building tools to actually come up with the workflows metrics.

And we touched on a little bit on the previous episode with Abi as well, right? About how survey can actually be simple to start with as well. And it’s so powerful. Don’t forget, simple doesn’t mean it’s less useful. But simple can also mean more powerful.

So maybe if you can touch on a little bit here as well how a survey can be done better, for people who wants to start maybe asking their developers. So any tips here?

Abi Noda: Oh boy. It’s funny because we do talk about how surveys are a good way to get started with measuring developer experience. And actually Nicole has written another prior paper where she talks about the trade-offs between system and survey-based metrics and also recommends beginning with surveys.

However, I now hesitate to describe it as easy. Cause as I currently speak to, I have four books on my desk about the rigorous design of surveys. So to take a step back, designing surveys that are, and I’ll use the terms, valid and reliable is actually a fairly arduous task.

And for listeners who aren’t familiar, and I wasn’t familiar, with sort of psychological psychometrics and survey design techniques before beginning this journey into developer experience. But when you’re designing surveys, it’s very easy to design surveys that don’t really produce accurate or reliable data. Because when you think about what a survey is really doing, what you’re essentially doing, and I’m gonna sound like a programmer here, but what you’re essentially doing is using the human mind as a tool to provide you data about a thing that you’re trying to measure. You’re using the human mind as a measurement instrument. And in order to do that, you need to have precision in the same way that we want precision and deterministic algorithms in our software and computer programs. We want to design surveys and survey items that behave in the way that we’ve intended for them to be designed.

And when designing surveys, the two things you’ll hear talked about are validity, for one, which refers to are you actually measuring the thing that you think you are measuring? And then there’s also what’s called reliability, which is, does your measurement instrument actually produce consistently the measurement you want over time and across different areas of the population?

That’s a crash course in survey design and Peggy I’m sure has a ton more experience she can speak on. But it is easy to get started with surveys, but it is quite difficult to do them in a way that produces accurate and reliable measurement at scale.

Margaret-Anne Storey: Yeah, I’ve tried to work with both kinds of data, and I think survey is actually much harder to design and get right than writing code to analyze and scrape data from GitHub repositories. That’s actually, I think, easier to analyze system data.

And so yeah, survey design is really hard. And as Abi said, there’s some great books out there. There are sometimes you can use the scales that are already out there. The problem with some of those is they may not be used exactly in your context. Or they may be so complex and ask so many questions that developers won’t answer them.

But there are certainly things that you can do and there are certainly ways to check whether your survey responses are reliable. But in the end of the day, having both kinds of data and then not blindly taking any kind of data without reflecting and discussing it is foolhardy, right? You need to go back to the people who answered your survey and ask them, here’s what I found. Does this resonate for you? Why are we seeing this? Ask for explanations, and why we’re seeing this in the data and the survey responses. And then they become powerful at guiding what to do next.

I think often with a lot of the dashboards that I see being used in industry to measure productivity that they measured it and it’s done. Look, I measure productivity. There you go. And now it’s yeah, but now the hard work starts in knowing what to do next.

And so surveys, we don’t talk about it necessarily in this table, but you can also use it for collecting open-ended answers, right? Which Abi has done lots of, and it’s those insights, right? The actionable, the insights on why something’s broken, not just why something’s broken, but how to fix it, is gold, right? That you will never get from system data, for example.

Henry Suryawirawan: Thanks for reminding us that actually the most important thing is actually create that communication channel, right? To actually listen for feedback. And you can ask open-ended question, you can ask all this multiple choice, which is probably difficult as you mentioned, right? I didn’t know that there’s a lot of theories about building surveys. Or you can actually get tools, like getdx itself, provides out of the box surveys, platform for people to actually fill in their feedback.

And also, I would like also to say that taking action is also important, right? Collecting data and getting dashboard is fine. But actually how do you improve after getting the surveys? And I think one thing you also mentioned in the survey, right? I know that since we started working remotely, maybe a lot of organizations and teams create a lot of surveys, right? People might have survey fatigue as well.

So please also bear in mind that people may not want to always fill in the form, but not seeing any actions being taken. So I think all this also come into the aspects about the perception part, right?

Margaret-Anne Storey: Yeah, really good point.

[01:00:10] 3 Tech Lead Wisdom

Henry Suryawirawan: I know we are running out of time, so I have one last question. Since I already asked Abi in the previous episode, I think Peggy this time is for you. So I call it three technical leadership wisdom. Think of it like, you have some advice for listeners here, maybe from your experience or expertise, what would be your three technical leadership wisdom, Peggy?

Margaret-Anne Storey: Yeah. I don’t know if I could claim it’s wisdom, but, maybe it’s just advice that I think technical leaders could take. So I think that there’s often that people think about software as code or features that are delivered to the end users. And I think thinking about code or thinking about software as the theories that people have in their heads is a better way of thinking about code. So no one person working on a project, even when they’re working on their own project, will understand how everything works, right? They’ll have incomplete models about how that code works. And so it’s really important to think about the theories that the different people that are on the team or other teams that they collaborate with, to think about those theories about how the code works, why it works the way that it does. But more importantly, what are the possibilities for how that code could be changed, right? Because any software that’s gonna be useful is gonna change. So thinking about, code as those theories in people’s heads. So thinking about people as part of the software is critical. And thinking about collaboration, which has come up a lot in our discussion today. This point about how people collaborate, how they communicate, how they share knowledge, how they help each other, how they exchange ideas and build on new ideas. This is absolutely critical, I think is the first piece. So don’t just think of software as code. Think about software as people and the theories that are in their heads.

The second thing, and this also came up in our discussion today. Sometimes I wish the term productivity would just go away in software engineering, but it isn’t going to. And I would even say the same with quality, right? That these terms, we use a lot of terms and we don’t define them. We don’t define what productivity means or what quality means.

I did that study that asked how developers and managers defined productivity and quality, because I’d sat in meetings and heard everybody saying if we do that, we’re trading off quality for productivity. And I’m like, nobody has defined what they mean. And I think Abi’s tool, the getdx tool helps with this a lot, having those conversations. In that study as well, we asked developers how did they think managers define productivity, and how did managers think developers define productivity? And what we found is that managers knew how developers thought, but developers didn’t know how their managers thought. And this shouldn’t be a secret. Particularly that developers, they do value collaboration, they do value helping others, but they don’t necessarily think their managers think that it’s valuable. But they don’t really know, right? And it’s this point that, maybe collaboration is invisible, but maybe it can be surfaced in some ways so it becomes important. So talk more about what you mean. Reflect more on these things.

And then the last thing is, and I’ve mentioned this as well, is recognizing that developer experience, a lot of it is, we’re human beings. We wanna have impact, right? And what we do, we wanna feel that the work that we do has value, that it brings value to our teammates or to our end users. And so recognizing that, making that impact visible, is really important. And it often takes some work, right? It’s often a lot of developers are shielded not necessarily from the impact that their tools have. And also supporting learning, right? So recognizing that learning is something that developers feel is impactful as well. So I hope that helps a little. I don’t know if it’s wisdom, but it’s just maybe some insights that are often at miss.

Henry Suryawirawan: Thanks for sharing these three. So I really love the second one. So I remember reading one of Abi’s newsletter as well about this. The different perceptions about productivity. And you mentioned about quality as well between managers and the developers, right? Sometimes what the developer think is quality is perceived differently than the manager.

So having it clearly defined, clearly communicated, or maybe even talked about, I think it’s really important so that we understand the same thing, right? Not just different perceptions by different personas or different roles within the company. So thanks again for that.

Maybe for people who love this conversation, they wanna dig out more. Is there a place to find online? Is there any forum to talk about it?

Abi Noda: Yeah. Both, yeah, myself, Peggy and Nicole, we’re all pretty active on social media. We’re on Twitter. We’re on LinkedIn. And yeah, me in particular, I’m pretty active on LinkedIn. Yeah, just feel free to reach out to any of us, as well as Michaela, the fourth author on this paper. If you have any questions or wanna talk more about this stuff.

Margaret-Anne Storey: Maybe we’ll have a blog post about it as well and just have some comments or something. It would be good to have some discussion, I think.

Henry Suryawirawan: And I also give a plug to Abi’s newsletter. So it’s always interesting that Abi digs out a lot of papers from the past history, right? And also coming out with the summary or insights. So I think those things are really great for people who love about all these theories and how to actually do a practical stuffs out of them. So thanks for that newsletter.

Thank you so much for this time. I really love the new paper and I really love this conversation as well. So hope you had a good luck in spreading this new insights to the industry.

Margaret-Anne Storey: Great. Thanks. Yeah, thanks so much Henry. Lovely discussion.

– End –