#103 - Software Development Pearls - Karl Wiegers

[00:01:20] Episode Introduction

Henry Suryawirawan: Hello, my friends. Welcome back to the Tech Lead Journal podcast, the show where you can learn about technical leadership and excellence from my conversations with great thought leaders in the tech industry. And this is the episode number 103. If this is your first time listening to Tech Lead Journal, subscribe and follow the show on your podcast app and on LinkedIn, Twitter, and Instagram. And if you’d like to support my journey creating this podcast, subscribe as a patron at techleadjournal.dev/patron.

My guest for today’s episode is Karl Wiegers. Karl is the author of “Software Development Pearls” and the Principal Consultant at Process Impact. In this episode, Karl shared some lessons that he has learned over the past five decades of his career, or what he refers to as pearls. We first discussed software requirement and its role for communication, and why it is very important to define the right software requirements. Karl then touched on the reasons why we cannot optimize all quality attributes that we desire in a software, and instead advised how we should define the quality attribute requirements. Next, Karl shared some project management pearls, related to work planning, dealing with estimates, and making commitments. Towards the end, Karl explained the relation between quality and productivity, using pain as a driver for improvement, and lastly, his ultimate pearl of wisdom.

I hope you enjoy listening to my conversation with Karl. If you find it useful, please help share it with your friends and colleagues who can also benefit from listening to this episode. It is my ultimate mission to spread this podcast to more people, and I really appreciate your support in any way towards fulfilling my mission. Before we continue to the conversation, let’s hear some words from our sponsor.

[00:05:07] Introduction

Henry Suryawirawan: Hi, everyone. Welcome again to the Tech Lead Journal podcast. Today, I have with me someone I’m really looking forward to learning from him. His name is Karl Wiegers. He’s the principal consultant with Process Impact. And he has written 13 books, one-three books. That’s really a lot of books out there. And today in particular, we’ll be talking about his latest book titled “Software Development Pearls”. The subtitle is actually “Lessons from his 50 years of experience”, so five-zero. That’s really a lot of years out there. So yeah, Karl, thanks so much for being in the show. I’m really looking forward to learning from you with all your Pearls in the book.

Karl Wiegers: Thank you, Henry. I’m happy to be with you today.

[00:05:46] Career Journey

Henry Suryawirawan: So Karl, your book has 60 Pearls. When I look at them, it’s really great. But maybe before we start into those, can you maybe start by sharing your career journey, mentioning about highlights, turning points?

Karl Wiegers: Okay. I have a slightly unusual background. But the age group that I’m in, a lot of people came into software from different backgrounds. We didn’t have a lot of computer science programs or software engineering programs in college. So I first learned to program in Fortran, of course, back in college in 1970. It’s hard to believe that was nearly 52 years ago, but it was. I didn’t go into software right away. I actually have a PhD in organic chemistry, which I’ve always thought was the perfect background for anyone in IT. Although not everyone seems to understand that. Well, one third of my PhD thesis was code. So I’ve done a lot of programming in a lot of different ways. After graduate school and postdoc, I began my career as a research scientist at Kodak in New York.

And one of the major turning points was around 1983, when I did switch over into doing software development full time for several reasons. But software had been my second interest anyway, and by then it was becoming my first interest. So I was a software developer at Kodak for several years, and I managed a small software group in the Kodak Research Labs. Then I moved more into business analysis, quality engineering, and process improvement.

And another major turning point came in 1996 when my first book was published, which was called, “Creating A Software Engineering Culture”. And that was a lot of fun. I always wanted to write a book, but I really didn’t have an idea. But some ideas came together then, and that turned into my first book.

And then another big turning point happened a couple of years later, 1998 when I left Kodak and started my software development consulting and training company, Process Impact. Since then, I’ve continued to write books periodically on various topics, including requirements, peer reviews, project initiation, consulting, product design, and even a forensic mystery novel. But for the last 25 years, mostly I’ve been doing consulting and training.

Henry Suryawirawan: So maybe some other time we’ll cover your forensic novel, but for today, we’ll be covering about your software development Pearls. As you mentioned in your career journey, you’ve been dealing with a lot of different areas, right? It’s not just purely software development. So you’ve moved into requirements, design, process improvement, culture. I think it’s really great to see all these actually covered in the book as well.

[00:08:07] Requirements for Communication

Henry Suryawirawan: So there are in total six areas, if I’m not mistaken. Starting from like requirements, design, project management, culture and teamwork, quality and process improvement. We may not be able to cover all your 60, but I think we’ll just take the important one or the interesting ones for you. So let’s start with requirements. I think this is very interesting when I look at the different Pearls that you have. We’ll just focus first on the first thing, which is the overarching objective of requirement development is clear and effective communication. So tell us more about this because yeah, we know requirements should have a very clear and effective way of expressing the requirements of the software. But can you maybe elaborate further more?

Karl Wiegers: Yeah, I always think of requirements as being all about communication. Software development is maybe half communication and half computing. But requirements is all about communication. So, I use the term business analyst to refer to whoever on a project is responsible for bridging the communication gap. They always exist between the customer domain, whatever kind of customers or users you’re talking about, and the implementer domain, whoever has to go off and then build some solution to meet some need or exploit some business opportunity.

So the BA is a role. It’s not a title necessarily, although many times it is. But it’s a role and in the central role of the BA, I think, is to facilitate exchanging knowledge about the requirements among all of the project participants. So it’s a really critical bridging function. And there’s no single way to accomplish this communication. People use different kinds of names for different kinds of information. If you’re talking about user stories in an agile world, if you’re talking about use cases or functional requirements more traditionally, it doesn’t matter. My principle here is that the developers need the same information to write the correct code properly, regardless of what you call it, regardless of what development life cycle you’re following. So I don’t think there’s anything special about agile requirements and calling things by special names.

In fact, there’s a lot of confusion over even what the terms used in requirements mean. So I find that when I start talking with people about requirements, the very first thing I have to do is define our terms so we can communicate effectively. For example, if I say business requirement, different people have different ideas about what that means. So we’re immediately at a communication disadvantage.

Another problem that we have is that we have multiple audiences for requirements information who have different needs. They want to see information in different ways, need different levels of detail. So the BA has to focus their requirements activities on getting a whole lot of information from various sources and presenting that in ways that let all these various audiences get the information they need to do their jobs effectively and efficiently.

There are two techniques that people sometimes seem to try for requirements work. Telepathy–reading minds–and clairvoyance, knowing things somehow magically. Those don’t work very well, but they seem to be the technical foundation for some projects. I also have another philosophy when it comes to communication here. I think the cost of recording knowledge is small compared to the cost of acquiring knowledge. And so, sometimes people aren’t very interested in recording information, or we don’t have time to write that down, or we don’t need to write that down, and I think that’s a terrible mistake because human memories are imperfect. They’re transient. They’re fallible. Have you ever had the experience, Henry, where you’ve had a conversation or a meeting with some people. You walk out. You think you’ve agreed on something. You think you have a common understanding and then a little bit later you realize, “Oh, we interpreted something differently. We remembered something differently.” Have you ever had that kind of experience?

Henry Suryawirawan: Yeah, almost all the time. Especially in this remote work. So you had a sync, maybe you chat a little bit, but yeah, it turns out the understanding is actually very different.

Karl Wiegers: And you don’t know what’s different, of course. So you just go off and do your work thereafter in good faith, based on what you thought you understood, and someone else thought they understood something different. So I think having a group memory, a documented group memory, by documenting requirements at an appropriate level of detail, gives you some shareable resource that can persist over time. And even if you did all have the same understanding when you walked out of that discussion or meeting, who knows what you’re going to remember in six months, when you have to go back and say, now, what were we doing here? Or I’ve got to fix that, or somebody requested a change.

So, sure, documentation can get out of date. No question. It takes some effort to maintain it. No question. But I think a shareable group memory, is the phrase I like to use for that, is a valuable asset to have. And then it’s up to the BA to work with their various stakeholders and audiences to figure out, well, what’s the best way to represent knowledge to meet your needs? When I talk about “writing requirements”, I mentally translate that into “representing requirements knowledge”. Sure, natural language text is the most common, but you can draw lots of kinds of pictures, visual analysis models. We can use tables or prototypes or photographs or mathematical expressions. Some people want all the details. Other people just want a high level view. So the goal of trying to figure out how do we communicate all that knowledge clearly and accurately, that’s really the essence of requirements work, I think.

Henry Suryawirawan: I like the two techniques that you mentioned, telepathy and clairvoyance. I think in the practical world, this is what tends to happen. Again, especially in the remote work, where you don’t actually sit side-by-side. Yeah, maybe you have a sync, you kind of like talk about it and then okay, people just split and do their own works, and actually they don’t sync very often. So sometimes this miscommunication happens. Maybe when you have a particular edge case, you don’t want to ask because you’re afraid that person is busy or whatever that is. So I think that tends to be a lot of this unclear and ineffective communication happening.

[00:13:49] Importance of the Right Requirements

Henry Suryawirawan: And I think one of the particular important things that you mentioned as well, right? It doesn’t matter if you have a very great team who can implement the code in the best way, but if they’re doing the requirements wrong, they’re probably, most probably, building the wrong thing. So tell us more about this cause of not having a good requirement and someone implementing it the best way they can.

Karl Wiegers: Yeah, that’s a real source of inefficiency. I believe everybody’s working in good faith. They’re trying to do the best job they can based on their knowledge and their assumptions. But you touched on an important point, like an edge case where maybe something comes up as you’re implementing a particular capability. And you think of an exception that we hadn’t talked about, or you think of a particular, as you say, edge case that we hadn’t discussed. How to handle that? You have to make some assumptions then, or you have to go chase down the information and somebody might be busy or not available or in a different time zone. Like you and I are what, 19 time zones apart, something right now. It’s hard to reach people sometimes. And so, you make assumptions, you make guesses and you might have to go back and fix it.

And that’s the problem. Because anytime you have to go back and redo something, anytime you have to fix something because of a misunderstanding or a piece of knowledge you didn’t have or something that changed, that’s rework. And rework is expensive. Or simply just having errors. I mean, we’re human beings. We’re going to make mistakes and we’re going to write something wrong or we’re going to be ambiguous in the way we say something and people interpret it differently from what we meant. So then you have to go fix it later. Well, anytime there’s an error, the quicker you can find that error and fix it, the less work you have to do to fix it.

That’s why it’s so important to, I think, do, for example, peer reviews of requirements. I think the highest leverage quality practice that software has available is effective peer reviews of requirements documents. Because errors that you find before someone wrote code are cheaper to fix than afterward. No matter what life cycle you’re following or how quickly you’re doing it. If you have to write the code twice, that costs you more than if you have to write it once. So that’s why I think it’s so important to not necessarily try to perfect the requirements, but to try to get them better than just some vague conversation and hope people can go fill in the blanks without any confusion or errors.

Henry Suryawirawan: So I also notice this kind of research when you find the errors closer to the right side in the value stream mapping way, in the production especially, it costs like really exponentially expensive rather when you do it in the more towards the left. That’s why this term called shift left. You try to reduce the error from the beginning. Thanks for sharing that.

[00:16:23] Importance of Definitions

Henry Suryawirawan: So you mentioned something about explaining about the key terms. Some people refer to it as the domain, right? So the domain terms, domain-driven design. Why having this shared understanding, maybe be it the terms, maybe be it the calculation or whatever, is actually very important for maybe business analyst, the product manager, and also development?

Karl Wiegers: It is. That’s why one of the tools that I think is a good idea is to simply create a glossary of terms. I was working with a client once and I was reading a requirements specification, and they were planning to outsource the development of this particular product. So I was reading this because when you’re outsourcing something, you’re basically mailing a requirements spec or something over to other people to do. Then it has to be better than if you’re sitting down side-by-side with someone where you can talk about issues as they come up and fill in the gaps.

So I was reviewing this for them and I found there were two terms that they were using that I thought might mean the same thing, but I wasn’t sure. And it turned out yes. Those mean exactly the same thing. In that case, let’s just pick one term and stick to it so that people don’t have to mentally translate it or wonder if they’re different. Because every time you do something like that where you’re using synonyms, or things that are close but maybe not quite the same or terms that mean the same thing, or mean different things rather, say in the business domain versus your technical world, every time somebody encounters one of those, they have to try to figure out, okay, exactly what does this mean? I’m not sure I better ask somebody or I’ll make a guess. All of those approaches are less ideal than just having a common vocabulary and understanding to minimize some of those miscommunications.

Henry Suryawirawan: And worse, if let’s say the development team treats it as two different things. They write it in two different variables or two different concepts in the code. The next person who comes, again, like what is this? And they build on top of that. So in the end, I think you have a very fun, wild way of expressing the requirements in the software itself.

Karl Wiegers: But if you might implement it two ways as two different things, and then somebody else comes along and says, “We can just refactor this. This looks like the same thing to me. Let’s just refactor it into one”. Maybe they really were different and you can’t combine them, and then you have a problem. This is what happens when people are hasty. Sometimes they don’t want to take the time to do this. They want to get to the real work of coding. And I think thinking is a part of the problem or a part of the solution, really. It’s an important part of what we go through and not everything in software has got to be code. Thinking can save you writing the wrong code or writing the code more than once.

[00:18:48] Optimizing Quality Attributes

Henry Suryawirawan: Thanks for covering this requirement, but let’s move on to the design. When you mentioned about coding, of course, design is one thing, right? One of the things that you highlighted to me is that you can’t optimize all desirable quality attributes. So I think quality attributes, some people call it non-functional requirements or these kind of terms. So tell us more about why we can’t optimize all quality attributes.

Karl Wiegers: Yeah. I think that’s a real important point. There’s functional requirements and there’s non-functional requirements. Some people don’t like the term non-functional cause it doesn’t tell you what it is; it tells you what it isn’t. I think quality attributes are what people are normally thinking of when they say non-functional requirements. There are really a lot of different dimensions of quality. There’s no single measure of quality that tells people everything they need to know about the quality of our product or what quality they want to see. Some of these quality attributes are external and visible to users like usability, reliability, security, robustness, and others relate more to the product’s internal quality, such as portability or maintainability or verifiability.

But the reason you can’t optimize all of those is because there are a lot of different kinds of relationships between them. So sometimes when you increase one of those desirable attributes, it increases another one. That’s good. If you increase reliability, you’re going to increase availability because the software is not going to crash as much. But if you increase other things, it might decrease a different attribute. And I think a good example is around security. I think we all agree that multifactor authentication is more secure than just using a simple login password, but there’s a usability penalty you pay for that because now the user has to take multiple steps, maybe take more time, certainly take more time. They might need multiple devices to log in. Sometimes I’ve had to use my phone so I can watch certain shows on streaming TV. I understand the security, but it’s a little silly, and it’s clumsy. So you hamper usability. You reduce usability because you’re increasing security.

And then other attributes don’t have much of a connection. But even within a certain quality attribute, they can be very complex with multiple dimensions. So, like usability includes both ease of learning and ease of use. So if you optimize the design for ease of learning by new users or occasional users, that might make the system frustrating to use for experts, because it slows them down, wading through menus, instead of being able to do something in a command-line interface.

So you have to balance those different quality attributes against each other. And that means you need to do enough requirements elicitation around those aspects of the requirements that you understand which of those quality dimensions are most aligned with achieving our business objectives. Which ones are most important to our most important customer classes, for example. And so, we really just have to decide what does quality means to your key stakeholders and then align everyone’s work towards those objectives. And if you don’t do that, if you don’t specify the details of these desired quality characteristics, if you don’t prioritize them so that the designers can make appropriate trade-off decisions, then you’re just lucky if you get whatever quality you hope to see in the final product.

[00:21:59] Specifying Quality Attribute Requirements

Henry Suryawirawan: So thanks for highlighting all these different trade-off balance. Sometimes when you increase one quality attribute, yeah, you may be lucky it increases the others, but most of the time it actually reduces the others. So it’s like an opposite effect kind of thing. If we can tie this to the requirements back, how do you specify these quality attributes in the requirements? Sometimes stakeholders, product people, they just tell yeah, you just build to the best quality, because they are really not aware of what quality attribute to think about, or maybe there’s just too many of them, so this -ilities, right? People call it -ilities. There’s so many -ilities out there. How shall we actually document this properly in the requirements itself so that it’s clear it’s effective?

Karl Wiegers: That’s an excellent question, Henry. And you’re right. The users will not usually spontaneously tell you what their quality expectations are. They sort of have tacit ones that they haven’t stated. And so, part of the business analyst job is to start those conversations so we can make that a part of our exploration of requirements during elicitation. The other thing that will happen is even if it comes up in the conversation, you’re likely to get a very simplistic answer. So you can’t ask a question like what are your availability requirements? Probably nobody knows exactly what you’re looking for or how to answer that question, or they’ll say the system’s got to be available anytime somebody wants to use it. Isn’t that obvious? So that doesn’t really help you very much.

So a BA has to ask more subtle questions, more direct questions at aspects of it, like are there certain parts of the system that’s more important to be available all the time versus others? Okay, well, that’ll give you an idea of priorities for things. Also, for example, if we have a system that requires periodic maintenance, I mean, sometimes I try to log into a website and it’s like, “Sorry, we’re busy right now fixing something, or doing our weekly maintenance, so you can’t get in”. Well, how long can you do that? What time windows can you use for scheduled downtime?

So by asking those kinds of more specific questions, rather than very general questions. You know, if you ask a user, “What are your interoperability requirements?”, they haven’t got the foggiest idea of what you’re talking about. But if you ask them, “Are there other systems you have to exchange data with or do you have to import data from anyplace to do your job with this system?” That might give you some information that is in the category of interoperability, but you didn’t have to even use that word.

Henry Suryawirawan: So the key thing that I took is that ask good questions, and always try to subtly dig deeper and deeper. Most of the times we do it for the functional part. So we ask, “Okay, how the feature should be? How does it work? What are the inputs? What are the outputs?” But not the same emphasis is taken for these non-functional attributes or quality attributes. So I think this is very important. Ask good questions and always try to dig deeper. Users, obviously, they’ll just do, you know, the best quality as you can, but we know that best is subjective, right?

Karl Wiegers: It’s contextual, depends on the situation and depends on who you’re asking.

[00:24:48] Work Plans & Friction

Henry Suryawirawan: That’s correct. So after we’d move to this design, let’s move to the actual work itself. You categorize this as project management. We have the requirements, we have the design, we have all this, so now time to execute. So one of the Pearls that you have is you said work plans must account for friction. So tell us more, what do you mean by friction? Why should we account for it?

Karl Wiegers: All right. So here, I’m talking about project friction, not interpersonal friction. That’s a whole other problem that we’re not going to talk about, but it’s a very real problem sometimes.

By friction, let me give you a good example. I was in a software team shortly before I went off on my own as a consultant. And I heard a conversation between the manager and one of the people in my group. That particular team member was providing a specific service to a project, and the manager asked her, “How much time are you spending on that particular service each week?” And she said, “Eight hours for this project.” And he said, “Oh, okay. That means you can work on five of them at once, then.” Because five times eight is 40, which is a nominal work week for many places. Even if nobody really works that number of hours, that’s what it is on paper. So that manager was making some assumptions that were not valid.

The problem is when you start getting people multitasking like that, you have inefficiencies cause people don’t really multitask, they task switch. And just like a computer, if you’re task switching too much cause you’re running too many things at once, it starts slowing down just because there are inefficiencies associated with the process of switching tasks where there’s dead time, basically. What happens is that maybe this woman could do her eight hours for this one project, and it’s time to move to the next one. You don’t just snap into a new way of thinking for that next project. It takes you a while to get everything in your brain, get all of your work materials in front of you, open all the right files so that you are ready to then be productive. It takes time just to load all it in your brain. The problem is that the more things you’re working on at once, the more time you lose to task switching overhead. And that’s why extensively multitasking people are not nearly as productive as you think they’re going to be. They just don’t have as many effective hours per week as it seems like they’re ought to be. It’s just the way people work.

Kind of related to that is the mindset–you’ve had this experience, I’m sure, Henry, all software people have– where you get into a state of flow. And that’s what happens when you’re working on something and you’re really productive. You’re making great progress. You’re having a good time. All of a sudden you look up and realize, “Oh, I forgot to eat lunch cause I was really cranking along.” You’ve had that experience, I’m sure. And so, what happens if that state of flow gets interrupted? You’re working along like that and then you get a phone call or a text message. And like Pavlov’s dog, we all check the phone immediately and see what the message is about. Or somebody calls you with a question. What happens to you when you’re in that kind of mode? You know what I’m talking about?

Henry Suryawirawan: Right, definitely. I mean, our brain works smartly, right? So when you have this interruption, especially when you think it’s urgent, you kind of like wipe the whole thing in your brain and then do the more urgent task. Maybe you pick it up, and then your context all changes. But then when you have to go back to the flow state, I think it takes a while. So the booting time is not as straightforward.

Karl Wiegers: No, that’s absolutely right. People have measured that, and it can take like 15 minutes or something to get back to where you were. Some people are better at multitasking than others. I don’t know if I just have a short attention span or what, but I’m pretty good at switching back and forth between things. Working on something for a little bit, then coming back and doing something else.

When I was a manager of a software group, I had a guy who was not as productive as he could have been or should have been. He was very bright, very creative, but he just couldn’t put work out the door as fast as I thought he should. So one of the things that we tried was having him block out chunks of time to try to eliminate some of this interruption-driven flow loss so that you get longer periods of productive work. And that actually worked pretty well. The rule is basically you don’t deal with emails. You don’t answer the phone for the morning. You can do that in the afternoon and get caught up on all the trivial things that you have to do. But if you block out your time so that you get bigger chunks of uninterrupted work, then you’re more productive.

So that’s, I think, the biggest source of friction and what that turns into when it comes to project management, as a manager or even as a team member, you need to figure out what’s your number of effective project hours per week on average. And use that number for estimating and for planning, instead of trying to say, okay, I got 40 hours to work this week. What am I going to do? Cause you don’t have 40 hours. You just don’t.

Henry Suryawirawan: Yeah. So you don’t account, for sure, like full eight hours a day. Because we chat, we take coffee breaks here and there. And I think what you mentioned speaks true to this concept called Lean. So limiting your work in progress, don’t multitask too much. Unplanned work as well. You have interruptions, that’s kind of like unplanned work, maybe incidents, on call, or maybe some managers asking about certain things in urgent. And I think another friction if I may guess, is about dependency where you are depending on some people or team or maybe some kind of work that has to be there, but it’s not available. Is it also counted as one of the friction in the project that you have to account?

Karl Wiegers: Yeah, that would be really a matter of creating wait states where you’re thinking, okay, I have to do these things. And as you point out, there are dependencies. I need this piece of information, or I need a module that somebody else was writing that has to be available before I can integrate it with my modules. If it’s not there, then you can’t finish your task. So I think it’s important to recognize those dependencies and build some slack time into your schedules to account for the possibility that not everything’s going to go exactly like clockwork.

It’s going to depend on the work environment because, for example, if you’re in a situation where you’re not only working on a new product, but you’re maintaining a previous product, or you have to go back and fix something from a previous iteration. Those kinds of interruptions have to be planned for, because that’s another form of friction that’s going to slow you down. And you didn’t plan for it, but you know that maybe on average, I’m going to have five hours a week that I have to deal with those kinds of things. So take all of that out of your planning for how you’re going to turn physical work hours into calendar time.

[00:31:03] Giving Estimates

Henry Suryawirawan: Speaking about effective hours, giving estimations, project management sometimes is all about estimating, right? You have these two very interesting Pearls about estimation, which I’ll read it here. Don’t give anyone an estimate off the top of your head. And the other one is based on what the recipient wants to hear. So I think software developers are most guilty of giving estimates right off the top of the mind and also what actually the requestor wants to hear. So tell us more about why this is not the right way?

Karl Wiegers: An estimate’s a best guess as to what’s going to happen in the future, based on what you know, and some assumptions. So maybe you’re walking down the hall and you run into one of your customers who says, “Hey, I’ve got an idea for some feature I want to add to the thing you’re working on now.” So they tell you their idea and you say, “Yeah, I think we can work that in. That’s not too bad. Sure. No problem.” So then you go back to your office and start thinking about it, and the more you think about it, the bigger it gets. There’s all these hidden connections. There’s all these ripple effects. There were all these unobvious depths of details. And then you realize that maybe it’s actually the opposite of what someone else agreed to do for a different customer the day before. So it gets bigger and bigger. But when you told the person that you bumped into the hall, “Sure. We can do this in three days.” That was an estimate, maybe off the top of your head based on what you knew at the time, but it sounded like a commitment to the person who heard it. They don’t necessarily know the difference between a commitment and an estimate.

So I think the best answer for any request about an estimate is to say, “Let me get back to you on that.” Instead of just giving them an estimate based on very little analysis and information, get the information, go think about it, do a more careful job of thinking about what it’s going to take to do that, then offer an estimate. Also, if you give a point estimate like, “Okay, that’s going to take 40 hours.” You can’t predict the future that accurately. You might be better off saying, “I think it will take between 30 and 50 hours.” The problem is people don’t like that. Managers in particular, people who have to write the check or allocate resources. They don’t like that kind of fuzziness, but that’s the reality. And I always tell people in my classes, I’m not always crazy about reality, but it’s all I’ve got. So I have to work with it. And the reality is we can’t predict these things precisely. So I think giving estimates in the form of a range is better. So that’s the kind of my point about the idea of not giving people estimates off the top of your head.

So there’s the other Pearl that says the estimate you give someone should not depend on what you think they want to hear. You mentioned that one as well. And I can tell you a story about that. I saw a conversation between a project leader and a senior manager. The senior manager asked, “How long is this going to take you to do this project?” The project manager said, “About two years,” and the senior manager said, “No, that’s too long. I need it in six months.” So what did the project manager say? “Okay.”

Well, what changed in those seconds? You know, he didn’t get four times as productive. He didn’t get more people. The problem didn’t get four times smaller. None of those things happened. He just said, “Oh, okay. Well, you want it in six months, so I’ll say I can do that.” That’s just a lie, basically.

I think knowing that someone’s not happy with your estimate should lead to a conversation, should lead to negotiation, and say, “Okay. What knobs can be turned to make this happen? Maybe my estimate’s wrong. What was your estimate? How did you come up with your estimate of six months?” The senior manager did not have an estimate of six months. He had a goal. The project manager in this case didn’t have an estimate either. He had a guess. So, if your guess and your goal don’t match, we’re going to have to work that out.

But I think actually doing an analytical estimate based on some thought process, some historical data, some depth of understanding is going to let you compare the differences maybe in assumptions that you’re making. But don’t just change it because someone doesn’t like it. That doesn’t change the future. You know, if I want to go on a picnic tomorrow and someone tells me it’s going to rain, I can’t just say, “Oh, well, I don’t want it to rain”. That doesn’t really change the future, right?

Henry Suryawirawan: Thanks for explaining these two different perspectives because sometimes manager or leaders have their own goals. And the software development, all we have is guess. We don’t actually know unless we have done it before. It’s like a repetitive work. We know the gist of the work. But yeah, most of the time we just guess.

[00:35:19] Pressure to Making Commitment

Henry Suryawirawan: Which brings us to the next Pearl, which actually is an extension of this problem. So you have given a range of an estimate, maybe, one month to three months. Obviously, managers don’t like that kind of ambiguity. It’s like a big range, anyway. And they kind of like put pressure. This often happens. I’m sure it’s everywhere. They’ll put pressure, but your Pearl says that no matter how much pressure others exert, never make a commitment you cannot fulfill. Okay. This is very tough. But tell us more about this thing.

Karl Wiegers: I think I learned this, and that’s actually a point. Let me back up just a bit. These Pearls didn’t come to me on stone tablets or anything. These are things that I just learned over my career, either from my personal experiences, or from clients that I’ve worked with, or from things that I’ve read and heard from other people. So I just accumulated these little bits of knowledge that I found helpful and relevant. The reason I wrote this book was to try to share some of those with other people, because you’ve probably noticed that experience is a great teacher, but it’s also painful and slow, and we learn a lot from mistakes that we made. I wrote this book so that people maybe don’t have to make all the same mistakes that I made and I’ve seen other people make.

One of the stories, I think, that applies in this case: how did I learn this lesson about never making a commitment you can’t fulfill? So one time I was leading a process improvement effort in a division of about 450 software and systems engineers in a big company. This was a very organized, systematic process improvement thing back when those were very fashionable. Me and my manager had a meeting with the senior manager of this division who was like four levels above me in the organization chart. He wanted to know what was our plan for achieving a specific goal with this big initiative? So I told him how long I thought it was going to take us to achieve that goal. And he was one of those guys that said, “No, that’s too long. I want it sooner.” He was the kind of manager, and there are a lot of managers like this, who will say, “Don’t tell me you can’t do it. Tell me what I have to do to enable you to do it.” And that sounds very collaborative, but in fact, it really wasn’t in this case.

So I explained to him how we came up with our estimate by looking at what other companies had done with a similar approach and stuff. And he was really pressuring me to commit to doing this goal in six months when I knew that there was absolutely no way we could do that. So finally, I said, “Fred, I’m not going to commit to that.” He just kind of looked at me. I don’t think anybody had ever said that to him before. He didn’t really know what to say. Now, I was a little nervous. Telling this guy all these levels up above me that I couldn’t commit to what he was requesting. But I knew that no matter how hard we worked, no matter how smoothly things went, it simply could not happen in six months. It just wasn’t doable. So I told him I couldn’t do it. So he said “Okay. You’re not going to commit. Well, now what?” So we talked about it, we negotiated some, and he eventually agreed to the schedule that I had proposed to him once I explained how we came up with that.

But my boss who was with me in the meeting, he was a little nervous. He went back, saying, “Karl told Fred he wasn’t going to commit.” But I think that’s the right thing to do, don’t you? I mean, if you can’t do it, what value is served by telling someone that you will? I don’t think that accomplishes anything other than temporarily making someone feel better and you feel worse.

Henry Suryawirawan: I think it also goes back to the psychological safety, this term being discussed a lot these days. If you don’t have psychological safety in the team, when the manager pressures you, yeah, of course, you’ll say, okay, for the sake of my job, I’ll just make that commitment at that point in time.

Karl Wiegers: And you have to make that choice. If it comes down to lying or promising something that you almost certainly can’t do, and might burn yourself out in the process. If it comes down to that or losing a job that you might actually enjoy, you have to decide what are you willing to do for that? But my personal philosophy is to under-commit and over-deliver. To me, that works. It’s not as aggressive as what some people might do, but on the other hand, I’m reliable. If you ask me to do something and I tell you when I’m going to have it done, it’s going to be done by then. You can count on that. And I think that’s maybe better than saying, where is it? Where is it? You said you’d be done. Why aren’t you done?

Henry Suryawirawan: In fact, maybe if you said, okay, I’m not gonna commit on that. Some sneaky leaders might find another person who will make that commitment and he will say, “Oh, okay. This person says, actually it can be done.” So then now you’re in a dilemma.

Karl Wiegers: For that phenomenon, it’s called the best liar wins.

[00:39:38] High Quality & Productivity

Henry Suryawirawan: So speaking about all these estimates and trying to come up with the agreement on the timeline and all that. Some people tend to associate time spent on the effort and the quality. So if you want to reduce time, okay, let’s just reduce the quality. But actually your next Pearl, your next category of the Pearl, is about quality. And one of the things that you highlighted is about high quality naturally leads to higher productivity. So tell us about this quality thing, because it’s always put as a lever that you can play.

Karl Wiegers: Right. In fact, one of the classic things that shows up in project management is the iron triangle. Have you heard of the iron triangle? Or the triple constraint is the other term that’s used for that. It’s shown in various ways, but it basically boils down to what do you want? Good, fast or cheap? Pick two. Recognizing that there are trade-offs and you just said, “Well, okay. You want it faster? I can do that. Do you care if it works? Well, cause if I go too fast, then I may have enough problems that it doesn’t work. How good is that?”

But the problem here is that everybody wants to be more productive. And if you look into why your productivity, either as an individual or as a team, isn’t as great as you’d like, one thing you might find is that you spend too much time on rework. I mentioned rework earlier. And that just is a productivity killer. Teams plan to get a certain amount of work done in a particular time. But then they have to fix problems found in completed work or maybe reallocate effort to repair a production system. A way to boost productivity, then, is to create high-quality software from the outset, so that teams can spend less time on rework, both during development and following release.

I learned this a long time ago. I was actually in junior high school in wood shop. My very first little project, we had this chunk of wood and we had to just use various tools on it, drill holes, cut it to size, that sort of thing. And if you got it too small or put the hole in the wrong place, you had to get a new chunk of wood and start over. It took me nine tries before I finally got it right. And I noticed that one of the kids in the class got it right in just two tries and he was done before me. He was going slower and more carefully. He wasn’t making mistakes, and so he didn’t have to keep starting over and fixing it. So that’s a lesson I’ve learned a long time ago.

So I hate rework. I hate doing something over that I’ve already done once. Some of it’s unavoidable. It’s the nature of what we do. But I think we just sort of accept it as a normal part of software development, and we don’t really focus on it as a lever, as you said, for increasing productivity by not having to fix stuff. The organizations that do track how much of their total software effort goes into rework can get some very scary numbers. Sometimes up to 40 to 50% of their total effort is doing things over that were already done once. And that’s why I think it’s so important, as you pointed out earlier, to push quality to the left. Let’s try to find problems earlier and quicker because that means we can fix them cheaper and not have to spend so much rework later on. Have you ever worked in an organization where people measured rework or thought of it as an explicit thing?

Henry Suryawirawan: It’s pretty rare. Very rare to have this kind of measurement and maybe we can use the analogy of the number of bugs found. But yeah, it’s probably not the same thing, right?

Karl Wiegers: Yeah. It’s certainly related because that’s where the rework comes from. But I think the thing we want to try to do is to minimize avoidable rework. There’s always going to be some. People are doing knowledge work. Human communication is imperfect. We can’t predict the future perfectly. But I think if we improve our initial work quality, we can minimize the avoidable rework.

One of the ways we can make this more visible is to, in our project planning, call out rework as a discrete task. A lot of times you might say, okay, we’re going to do this and we’re going to build something and then we’re going to integrate it and then we’re going to test it, and then we’re going to move on. Well, that’s never been my experience. My experience has been you build it, you integrate it, you test it, then you fix some stuff, then you retest it and eventually you move on. That we sort of bury rework as part of that quality control activity of testing or review. I think it’s better to make it a specific explicit task. And then you could actually measure how much time did we spend fixing things after the test or fixing things because of problems found during a peer review. That gives you an idea of how much rework you’re actually doing, and then you can ask yourselves, is that acceptable? Is that a reasonable amount of our effort? Or should we say, “That’s too much? Let’s try to find some ways to cut down on our rework.”

I did that in a group once. We actually measured for several years how we spent time on projects. How much time on requirements work and design and coding and testing and maintenance and so forth? We didn’t like our initial levels of bug fixing. And so, we focused on techniques that could help us do a better job of producing initial high quality software. Within about six to eight months, we reduced our defect correction rework from 13.5% of our initial work to 2%, which I was perfectly okay with. There’s always some, but 13 was higher than I liked.

So I think making that visible, making it explicit, trying to measure it, then at least lets you know what’s going on, so you can decide, is there some way that we can do a better job of initial quality to do less rework?

Henry Suryawirawan: So this also comes back the lean principle where you have value stream mapping, and maybe you will analyze these wait times and also rework. So that’s actually one part of the value stream mapping exercise. And yeah, reducing waste. We all agree that we should reduce more waste, more rework. But I think I just want to highlight one other Pearl that you have about this is that organizations never have time to build software. So we never build software, but we always find the resources and time to fix it later. I think that’s one thing for us to ponder about.

[00:45:16] Pain as Improvement Driver

Henry Suryawirawan: So let’s move on to the process improvement. You mentioned if we want to improve, the best motivation for changing how we work is pain. I think some people learn from the hard lessons. So tell us more about this pain.

Karl Wiegers: They do. I think that’s true in life. In general, the best motivation for changing anything we do is pain, which can come in many forms. And here, I’m talking about the real pain that happens on projects because of quality problems or process shortcomings or communication problems, those kinds of things. I’m not talking about artificial pain that’s externally induced by managers or customers who are demanding the impossible. That can be painful, but that’s not in itself terribly motivating.

Some examples of project pain that can be motivating are failing to meet delivery schedules, releasing products that have more defects than you think are acceptable or are missing functionality, delivering products that don’t satisfy the customer needs. Or maybe you got burned by risks because nobody thought about the risks and figured out how can we mitigate that risk and then it turned into a problem and you paid some price for that. So when you experience those kinds of pain, that should make you say, “Oh, that wasn’t very much fun. What can we do differently next time to try to avoid that unpleasant situation?” And so that, to me, is the driver for process improvement.

Give an example. I worked in the Kodak.com group. Kodak used to be a very large company. Now it’s not such a big company. In the late 1990s, I was in the Kodak.com group that ran all their websites, and they had a configuration management problem at that time. They had two web servers that were almost, but not quite, duplicates. They were having all these configuration management confusions cause they weren’t too sure exactly what to trust, and what was most current at any time. They also had a huge backlog of change requests, because there were a lot of people wanting websites at that point. So I was there in a process improvement leadership role, and I helped introduce a change request system and better configuration management practices that reduced the pain coming from that.

And the thing I found is that in this group, with a bunch of very smart people working very hard, very fast, sort of the leading edge kind of stuff now, they were very receptive to those process changes. A lot of times, people in fast moving groups are, “We’re not interested in process. We don’t have time for process.” But they felt the pain, and they saw the benefits. They understood that the processes I was talking about weren’t barriers to getting work done. They were structures to help you get the work done better. And they saw that, and they were very receptive to it, and it really made a difference in helping them manage all this work that they were trying to do.

So I think if you’re trying to get people to change what they’re doing, and they’re all busy working on their projects, then you have to make this pain visible so that the promise of reducing the pain outweighs the discomfort of making the changes. Because making changes isn’t that much fun, no matter what we’re doing. Whether you’re trying to lose weight or improve your diet or increase your software productivity, it’s never fun, but you sort of have to do it.

And one of the problems is– I’m curious if you’ve run into this–that some of these kinds of project pain aren’t always visible to everyone who’s affected by it. So maybe a development team is taking certain approaches, and that causes problems for whoever has to maintain the software later on, but that doesn’t have a feedback loop back to the development team. So they may not even know that what they’re doing is causing problems. Have you ever seen a situation like that?

Henry Suryawirawan: Yeah. This is the syndrome like we always done it this way. So sometimes you are into the things, you never see it like a fish in the water, so to speak. So you never know that actually you live in the water, you live with the pain. But yeah, you need to take a step out and find the big picture. Okay, these are the pain things that we have. But sometimes when we are in the gist of making things work and implementing stuff quick, we don’t always realize these pains.

Karl Wiegers: I think if you’re in a leadership role trying to steer an organization to better ways of working, you want to look for the points of pain, because those are the things that people can usually agree upon once they are made aware of them that, yeah, we can do better than that. That isn’t that much fun. Let’s find some better ways to do our work.

Henry Suryawirawan: One of the other things, another podcast episode that I have with Jardena London, she also mentioned about this making changes. So you want to ask people to change, you better stop the pain first from bleeding more, right? Sometimes when we all introduce this change management, you just introduce more tactics, more strategies, but you’re actually not necessarily fixing the pain that the team is currently having. So you’re introducing more pain, so to speak.

Karl Wiegers: And there’s always a learning curve you go through whenever you’re trying to do something new or different. There’s a learning curve where you take an immediate productivity hit as soon as you go take a class, or you adopt some new methodology, or you buy some new tool. Whatever you’re doing is different, immediately you have a productivity penalty because you have to figure out all right, how does this work? How do I apply it in my world? And it takes you some fumbling around. But then eventually, if you can push through that, you’ll get to a point where you’re being more productive and hopefully less pain with the new technique. But the change itself can be unpleasant. People have to recognize that there’s a short-term performance penalty in our work while we’re going through that transition.

[00:50:25] Ultimate Pearl

Henry Suryawirawan: This is a good segue to summarize all the Pearls, right? The ultimate Pearl, the Pearl number 60. So there are 60 of them. The last one you mentioned, you can’t change everything at once. I think people know about this, but sometimes they live in fantasy. They want to change everything all at once. So tell us more about this.

Karl Wiegers: It’s interesting, because sometimes you do have groups, and I’ve worked with some like this, that were very enthusiastic about process improvement. They really saw the benefits of making improvements in whatever areas weren’t working as well for them. And I remember a group once with about 20 people in the team who were like that. They decided to work on seven different change initiatives at once. But the problem was, they were spread too thin, and they didn’t really make much progress on any one of them. It wasn’t clear which were the top priorities: let’s do this first, let’s do that next. And so, they just tried to do them all at once kind of a broad front strategy. As a result, they didn’t make much progress, which is frustrating. You know, I think you only get two tries in an organization to make those kinds of changes that fail, and then people aren’t interested anymore. It’s like, I guess we can’t do this. So I think what’s important then is to focus your change efforts in the places where you’re going to get the maximum benefit for the investment of effort to make the changes.

And there’s an operative term that I learned a long time ago that I love about process improvement, which is: “Gentle pressure relentlessly applied”. In other words, you should all, as individuals, as teams, as organizations, you should always be spending part of your time learning how to do things better. I’ve never known anyone who could honestly say I am building software today as well as software could ever be built. I certainly can’t say that. I don’t know anyone who could. And if you can’t, then you should say, all right, let me try to find better ways. Let me always spend some of my time trying to improve the way I do my work in one way or another.

So related to that is the idea of doing team retrospectives. This is a hot topic in the agile world, which I think is great because retrospectives are such an important learning opportunity for groups to say, “All right. Let’s reflect on what happened. How did it go? What parts went well? Let’s do that again.” But parts didn’t go so well, and let’s not worry about blaming people. That’s not the point, but let’s just learn. Let’s try to see if we could do those better the next time. What happened that we still don’t understand? There’s some learning opportunities there. What happened that surprised us? Maybe some risks came up. We haven’t thought about before, so you add those to your risk list. So on the next project you ask yourself, could that happen again? So a retrospective is an excellent learning opportunity that helps you to kind of gently try to keep changing things just a little bit at a time.

Henry Suryawirawan: Thanks for emphasizing this. I think focus is really key. No matter whether in projects, but also in your individual life. Focus is definitely one thing that I think modern people, right, we have this problem of focusing. That’s just too many things to consume, too many things to learn, too many things to watch, and whatever that things. So, thanks for sharing this.

Karl Wiegers: One of the problems with the focus, we are so busy, we’re working on so many things, there’s so much pressure, everybody wants things yesterday, is that if you don’t take the time to learn and improve, you have absolutely no reason to expect the next project to go better than the current project. You could hope, but you have no reason to believe that’s going to happen.

Henry Suryawirawan: So it’s a magic if you expect that to happen by itself. So thanks, Karl. It’s been a pleasure talking about all these Pearls. So for people who want to learn more about the other Pearls, there are 60 of them, like I said. They are lessons learned by Karl in his 50 years of software development experience. So really worth to check because, I mean, where else can you find these 50 years of experience densed into one book and you don’t have to repeat the same mistakes. I think that’s the benefit of reading this book so that you don’t actually redo the mistakes that maybe Karl did in his previous work.

[00:54:09] 3 Tech Lead Wisdom

Henry Suryawirawan: So Karl, due to time, unfortunately we have to wrap up pretty soon. But normally before I let you go, I have one last question that I always ask to all my guests, which is to share this thing called three technical leadership wisdom. So think of it like an advice that you want to give to listeners out there so that they can maybe reflect and maybe do it better for their own sake.

Karl Wiegers: Okay. You kind of led right into that with your final comments there about learning from what we’ve done before. And the way I think about it is that, you don’t have time to rediscover everything that all the software people before you have already learned. So I think it’s important for people to read the literature, respect the lessons of the past, adapt established effective practices to your new reality instead of relearning these same lessons painfully. Sometimes there’s a tendency, I think, especially among younger people to throw out anything old, especially in software. It’s such a young field in itself and moving along. People don’t say, “Oh, that was old chemistry. I don’t remember it. That doesn’t apply anymore.” Yeah, it does. There’s a lot of stuff that still applies that we’ve known for a long time in software, and maybe you need to adapt it in some way, but the ideas still apply. And so, respect the body of knowledge that we have.

Another piece of wisdom gets back to the “you can’t change everything at once idea”, but you can change something all the time. One way I suggest people do this is, on every project you work on, pick out two areas you want to get better at. They could be specific technical practices, or softer skills, or anything else that you think would add value to your project work and your career. Some of the things I chose at various times in my career were things like build management or unit testing, analysis and design modeling, writing requirements. Things that I said, I need to learn more about that. So then spend a little bit of your time on your project learning about those areas and trying to actively apply them to get some experience with them on your current project.

And the third bit of wisdom that I think I like to pass along is before you decide to adopt any particular solution that you think is going to give you great new results, cause you read an ad somewhere or that’s what the person at the conference said, make sure you understand the real problem. I’m a big fan of root cause analysis. Let’s say you’re tempted to adopt some new methodology. I’ll call it Method 9 that’s supposed to give you great productivity or higher customer satisfaction or some other fabulous goodness. Before you switch to Method 9, first ask yourself, why are we not already having great productivity? Why do we not already have higher customer satisfaction than we do? So once you’ve done some root cause analysis, then you can decide whether the solution you’re considering is actually going to address the right problems. In other words, if Method 9 is the answer, what was the question? So I think it’s important to do some root cause thinking before you choose any new way to work. Then you can make sure you’re choosing the right one.

Henry Suryawirawan: Wow. I really like the last one. It’s really wise to have that. So because, yes, in software development, especially we tend to follow this hype-driven development. Always chase the latest hype, latest technology. Be the next Google or whatever that is. And then try to apply that in our working environment. Okay. Sometimes it’s not the same problem. Because of that, you introduce more problems for the team and the company itself.

So thanks, Karl, for your time. For people who want to learn more about your work, maybe other books, where can they find you online? Or where they can reach out if they want to continue this conversation?

Karl Wiegers: My personal website is KarlWiegers.com. My company website is ProcessImpact.com, and there’s information at both of those about all of my books on various topics. There’s a lot of articles. I’ve posted a lot of articles both on the ProcessImpact.com site and on Medium.com. I have 85 or so articles up there on a wide range of topics. So those are all places where people can learn a little bit more.

The other thing you can do at KarlWiegers.com that may not be obvious. My main hobby is recording music just for fun at home. I play guitar. I’m not a good singer. I’m not a great guitarist, but I don’t let that stop me. So I like to record music and I have about 50, I think 51, songs now at KarlWiegers.com that I’ve recorded. Just me doing all the parts, maybe a little help from some friends once in a while. 18 of them are songs that I wrote myself and recorded, and the rest are all covers of other songs. It was just a fun hobby.

Henry Suryawirawan: For people who want to listen to new music, instead of Spotify, go to KarlWiegers.com and make sure to check out these original songs that Karl has produced. So thanks, Karl, for your time. I’m really learning a lot from your Pearls. I’m sure people would also learn something if they read from the books. So thanks again for your time.

Karl Wiegers: Thanks very much, Henry. It’s been a pleasure being with you today.

– End –