#213 - Moldable Development: Explain Systems & Make Better Software Decisions - Tudor Girba

[00:01:41] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I’m very excited to have this conversation. We are going to learn something new. A new concept called moldable development. So I have Tudor Girba with me today. So very excited to have this conversation. Welcome to the show, Tudor.

Tudor Girba: Well, thank you for having me.

[00:01:57] Career Turning Points

Henry Suryawirawan: Right. Tudor, maybe before we start diving into moldable development, I’d like you to maybe tell us a little bit more about yourself by, you know, sharing some career turning points that you think we can learn from you.

Tudor Girba: Okay. Well, so I’m presently the CEO of feenk. feenk is a company that is focused on modernizing legacy systems. We exist since about 10 years. And the reason feenk exists is because we wanted to have a way to fund the research that led to what today is called moldable development. There’s interest in researching the space of how do we reason or understand systems and make decisions about systems. Dates back to a couple of decades, in my case. So I started with my PhD and work at the University of Bern and postdoc then. So my research area was software analysis, software evolution, mining systems, visualizations. So I was building quite a number of tools and there was an extensive platform at the time that helped us prototype analysis very fast and lots of them.

And at some point, I remember writing my n-th paper, starting with this sentence that developers spend more than 50% of their time reading code. And then I realized that the oldest reference on the subject that I knew of was something around, was from around 1979, which was about three decades back. So I was wondering, hmm, if I can state exactly the same problem three decades later, is it possible that we are asking the problem in the wrong way or looking at the problem in the wrong way? So I started to try to look at the problem differently. And then what was worse was then I looked at myself. So I was a researcher, I was building lots of tools that were supposed to automate and help developers understand their systems better. But when I looked at what I was doing, I was reading code for more than 50% of my time. Obviously, something was not quite right. And so I tried to solve problems afterwards.

So I left academia to try to pursue this idea, finding a different way to look at and how to improve decision making in software engineering. And so we knew how to build tools. We did have this extensive platform already in place at the time. It was called Moose. It was one of the largest platform is for analysis research at the time. Several dozen PhDs worked on it and lots of master students and so on and several universities. So I was the main architect of it. And so I started to do consulting and tried to solve actual problems. And when I realized I was looking for the recurrent pattern, cause I didn’t know exactly where to start from. And I was looking for the recurrent pattern. And then I realized that the only recurrent pattern was that in order to be meaningful for a specific system or specific problems in a specific system, was that I had to tweak my tools. I had to change my tools. That was the recurrent pattern over and over again.

And that’s when this other observation came into play, which is the software is highly contextual. Which means that we can predict classes of problems people will encounter, but we cannot predict the specific problems that people will encounter. And so, as a consequence, we cannot provide clicking tools as solutions. Or we can provide them, it’s just that they won’t be effective. Because the clicking, a clicking tool makes the problem in the click. So if I can’t know the specific problem, I’ll be answering the generic problem. And so the rest of the energy to actually go and answer a specific question about my system will have to be filled somehow, and it typically happens manually. So the alternative is to say, well, what if we can build a tool after we know the problem, which means during development.

So that was the idea about 15 years ago, and ever since I tried to validate, or actually we were looking for the contrary example. So me and my colleagues, that’s what we’ve been doing for the last 15 years, searching how is it possible? Is it possible to tackle every single development problem through a custom tool? And then is there a counterexample when building a custom tool is not desirable? We haven’t found the counterexample yet.

Henry Suryawirawan: Wow, I think you shared very, very interesting facts, right? Let’s start with the first thing that you shared, right? You mentioned that 50% of our time, I’m also kind of like developers, right? I think we spend more than 50% even now with AI, right? Because AI suggests more code than we wrote. We are reading even more, right? So I think reading code is definitely the main job of a developer and not just that actually to understand how the software is written. Because if your software gets bigger, right? You won’t be able to understand just by reading one file or a few files, right? You have to read maybe the edges of those files that you’re working with.

[00:08:29] Understanding How We Read Code

Henry Suryawirawan: So maybe tell us this problem that probably we all experience it, right? But somehow we kind of like what you said, right, use this generic tool, clickable tools, IDEs, to actually understand the code. Tell us why this is not really optimal. Like what kind of examples that you can give us why this thing doesn’t work for longer term?

Tudor Girba: I think that’s, a nice question. So there are very interesting studies over this going back a couple of decades measuring what developers do. And they all show that this, the amount of time we spend trying to figure the system out is way up, beyond 50%. Now, I wasn’t satisfied only with that. So I actually went and directly talked with more than like several thousand people, several thousand developers. And I asked them directly in various situations. And I asked them, do you agree with the statement that you spend more than 50% of your time reading code? Pretty much everybody says yes. So I was at GOTO a couple of months ago and there were like 500 people in the room. And I asked this question and then everybody said yes. But then I asked another question. When was the last time you heard developers talk about how they read code? So not about the code that they read, but about how they do the reading. It’s a weird question, right? What’s there to talk about, right? And it’s, nobody talks about it. It’s not a subject of conversation.

Now this is kind of interesting because, you know, you were talking about, you were asking, why is this suboptimal? If you don’t talk about something, it’s not explicit. If it’s not explicit, it has never been optimized. Now normally, let’s say when you’re searching, when you’re trying, let’s say you have a running system and you’re trying to optimize performance. You profile your system and you find like a big chunk, like more than 50% of the performance, in a single place, in a single piece of logic. And you looked at it and you look at it and say, wow, we’ve never optimized this algorithm. Now this is like a super great news, because now I can optimize a great deal of it because there’s a very large promise of optimizations possibility right there.

The same thing happens with activity. Like literally when we don’t talk about what we do for more than 50% of our time, that’s the single largest expense we have. There’s nothing larger than that. It basically means like we just, we’re going to optimize for the tiny bit. So we have this humongous amount of work and we’re just going to optimize this little tiny thing that we do.

[00:10:43] Software Engineering is a Decision-Making Activity

Tudor Girba: And this is by the way, right, talking about AI, that’s what happens. Like people just say, oh, what are we going to use AI for? I’m going to generate more code. So then, of course, what happens is the problem just becomes even larger. So that’s the first order logic, the first reason why we should actually start talking about it. We don’t even have to talk about the solution, right? This is just such an obvious place to start optimizing. So that’s basically where I started from 15 years ago, to look at it as a subject of conversation.

And so now what, how else, you know, can it be more optimal? So, for example in the AI world, right? We say, oh, we have this. Like we see these demos, right? It’s everybody talks about how I generated an application. Of course, the application is only 10 screens or so. But it’s basically you’re bombarded by these, you know, walls of text. And people say, oh, look, so great. It’s like, what? Like, how do you know? Because it’s just, you just have something there. So you have to go and read it, as a way to reason about it. But in fact, that’s the thing. You don’t have to read it. Because the reason, the question is why do you read code? Well, you read code because you want to understand enough a system to be able to make a decision. So a situation around the system to be able to make a decision.

Now from this, the activity is actually decision making. Now, given that software engineering is primarily like this is what we do most of the time, an AI just makes the problem more obvious. The problem existed at scale well before AI. So in any organizations we meet, people are drowning in software that they cannot move. That they cannot understand and change. This problem exists for a long time. We’ve been able to optimize more and more the thing that we talked about which was how do we construct systems. So we got better and better at producing more and more code, faster and faster and faster. Now to the point in which we generate some, I don’t know what code, and we’re gonna just say, oh yeah, put it into production. So this one got better and faster and faster and faster, but this other problem just became larger and larger and larger. And we’re not talking about it. So we definitely should be talking about.

The first thing to say here is that given this we spend so much time trying to make decisions, what if we just look at software engineering as a decision making activity about an ever changing system that is much larger than if it’s in our heads rather than a construction activity? Now when you change the point of view, like lots of things change. Lots of things that were important before, all of a sudden become much less important. Because now you have a different perspective to look at it. So this is the first thing.

[00:13:19] Reading Code is a Suboptimal Activity

Tudor Girba: So from that point of view, reading, for example, is just a strategy. It’s just a tactic, actually, of extracting information from the system. Right, now from this point of view it becomes obvious that actually this is the most manual possible way in which we can extract information from our system. So you see like once we make the subject obvious and we started talking about it, we do like two or three steps and we just find ourselves in a rather, like, what are we doing?

But then it gets even crazier, because if you think about what’s an enterprise system doing, for example, right? So all enterprise systems, they automate someone’s decision making specifically by preventing those people to ever see the raw data. So we take the data, we import it, we model it, and we present it in a way that other humans can make much faster decisions about humongous amounts of data without ever seeing the data. So who possesses the skill of transforming, you know, raw data problem into decision making problem? We do. Like software people do. So when software people have their own problems, what do they do? Well, we fall back to the most manual possible way of doing things. It doesn’t make any sense.

So, but the cool thing about this is, um, So there are a couple of things here. One of them is that we have a large opportunity for optimization. Because you have these like very large costs. It was like obviously we’re doing something that is so manual. So it’s like very, very good news. Because with just a little bit of investment in that space, probably can put you ahead of things. But the other thing is that there’s a significant, probably, you know, like this is just small investment can lead you to a significant short term type of benefits. But it’s more very likely that you can have layers of that. So just on this thing.

But the other one is we shouldn’t just be looking at the cost. So when we say, this is the amount of time we spend trying to gather the information, we should be also looking at what do we do with the result of that activity. Because the decisions that we make as a consequence of reading the code are much more costly than the time we spend actually reading the code. And this is where it becomes really exciting.

So, today, for example, it’s actually very rare to find a team that is able to provide an accurate depictions of their system. So let’s say, you know, an architecture diagram. People will be presenting architecture diagrams. But then I always ask them, so like, usually those are manually put together. So I asked them, would you bet your salary that the diagram is actually real? And nobody would take the bet. Because it would be, everybody knows, yeah, but it’s not expected to be actually that real, right?

And then the question is, well, why not? So if you take this as the basis for any decision making, shouldn’t it be real? And put it the other way around, if I’m in an engineering space, right? I’m not betting on, you know, what’s going to happen tomorrow. I’m in an engineering space where I’m trying to build systems that are kind of reliable and are meaningful and they produce some value. So not knowing what the current system looks like, probably not a good variable to have. So that’s what we’re saying is that while there is a systematic way to know about your system at much lower cost than it happens today. And this then has far reaching consequences.

[00:16:44] Moldable Development

Tudor Girba: This is the essence of moldable development, is we’re saying we’re going to take this time that we spent today or the developers spent today reading, and we’re going to take some of it, some of this effort, and we’re going to build tools through which we read. And then we’re going to make decisions on that, but these tools will provide an unmediated lens through the system. Because often, people say, shouldn’t we trust, you know, technical people that they know what they’re doing? Of course, we should trust technical people. There are some problems that shouldn’t be decomposed or seen as trust problems.

So an example that I always give is, let’s say that I’m interested in what’s the temperature outside. Is that a trust problem? Should I go and ask someone and trust that someone? Or is it just should I go and look at the thermometer which is outside my window here? And if I look at the thermometer, which is outside my window, does it mean that I’m not trusting the person next to me? No. Because that is not a trust problem, right?

So now, the issue with reading is that nobody can read the whole system. So if nobody can read the whole system, it means that most information that we will be basing our decisions on are going to be inferred. So beliefs, basically. So, the consequence is that at some point, your beliefs will be significantly, you know, apart from the actual reality of the system. And you’ll find yourself dancing around the fire to make rain happen. So it’s really like that.

So, for example, we’re writing a book with Simon Wardley, which tries to summarize what we’ve learned about moldable development. And one of the case studies that we’re showing there is an actual system. It’s an important data pipeline in a significant telecom company. And their issue was that this pipeline was too slow. So everybody knew it all the way to the C-level. They had a large initiative demanding that, well, we should improve the performance of this one. But like significantly, let’s say an order of magnitude or something. This is what they wanted. Now they worked on it for three years. And after three years, the performance was exactly the same as it was three years before. And so it was like, somebody looked at the problem and says, well, how come?

So when we looked at the system, we asked them, well, could you draw the system for us? And then they drew four boxes and said this is how the data pipeline looks like. So the way we always look at the problem like that is we’re going to build a tool that shows us that system, kind of like data science if you want. Like we start from the problem and then build a tool specifically for that problem. In their case, they had like custom made technology that was doing transformations and the local platform that didn’t have any kind of analysis tools and so on. So we had to build interesting analysis tools so that we can recover some sort of data lineage.

But while in this process, very soon, we went back and said, well, you know, like the output from the first box doesn’t seem to match the input for the other ones. And then we asked, so are you sure you don’t have another thing here in between? And they discovered the whole system. They were not aware that it was there. And it has nothing to do with the ability, like engineering necessarily, abilities directly. Because these were like good engineers. The problem is that when you work in teams and silos, right, and things move between, the most interesting problems are always in between the things that you already know. And so if you’re not prepared to see those inter spaces, you’re gonna miss the interesting opportunities, right?

So the issue in their case was they could enumerate four things when they had actually five. But if even at this large coarse grain level, you’re not able to see the whole system, you’re basically in no position to optimize anything. So it doesn’t matter how much effort you’re going to spend, because the problem is not, oh, are you able to change? The problem is I don’t see the system in the first place. Even if I’m convinced I’m actually understanding my system, I’m not seeing it. Once that’s the case, that’s the first problem to solve.

And I would say, okay, but this sounds like such a ridiculous case. But you’d be surprised how often this happens. Even in places where, which people say, Oh, I actually have a much better grasp of my system. You go and ask a few couple, like more detail, go two or three levels deeper, and you’ll find yourself in the place where people draw something on the whiteboard. Whenever people draw something on the whiteboard about the existing system, they depict their belief about the system, rather than the system itself. This is really important, because if you can’t distinguish between these ones, you’re going to find yourself being in a make-believe space. And then your actions will have no impact on reality.

Henry Suryawirawan: Yeah. So I think so many interesting things that you mentioned just now, right? So I think one major insights that I could listen from what you just said, right? Software development is actually kind of like a decision making process, right? And I think, through writing code, you know, through syntax, through whatever programming language that you use, it can be suboptimal, right?

Because whatever that we want to create, we are kind of like limited by the tools and the kind of like the thing that we express ourselves in, right? And I think we all know as a software developer, once you get to write your code in a longer time, it’s very, very difficult to actually understand what is going on. Especially, it can be very abstract, right?

And so many people working together in the team, some are new, maybe some have left. Maybe even like they don’t talk to each other. So there are so many different ways of how software is being written. And I think this is going to be a challenge, right? And plus, you may think that AI will solve it, but AI, you know, could also hallucinate by reading your code, right? So some of these definitely is suboptimal and I can really understand because now I’m currently, even though I’m currently working in a small startup, right? We have written code for, I don’t know, the past few months. There are still things that I think I don’t fully trust myself knowing the system end-to-end, right?

Just like what you mentioned, if we’re asked to come up with an architecture diagram. Yeah, we can kind of like draw high level picture, but we are never sure because the code is the fact right? So sometimes the code doesn’t reflect what you believe, like what you said.

[00:22:47] The Challenges with Legacy Systems

Henry Suryawirawan: So I think this also leads us to another thing that you mentioned in your writing with Simon Wardley, The Problem with Legacy System. Legacy here, I think, doesn’t just refer to old, really old systems. But software that you wrote maybe a few months ago can also be categorized as legacy. So tell us this legacy problem, why is it going to be a big challenge for us?

Tudor Girba: So we’ve internalized this idea that once you write the systems, we can’t really make sense of them anymore. Not really. So in rare occasions, when there’s actually like people are very, like, very joyful, but the vast majority of situations of the reports out there. So we, for example, in the book, we list, I don’t know, a paragraph worth of all sorts of different kinds of reports from that we took from industries, blog posts, and reports here and reports there. Of course, those are not necessarily super systematic, but they all point to exactly one problem. Like legacy is everywhere and it’s very expensive and it’s very hard. Like all these reports saying, oh, 70, 80 percent of, or whatever the number is of the migration projects are failing, right.

And that’s really like the reason, if you look back into why this is the case, it’s because we don’t see the system. The interesting thing is, as soon as people see the system, they have no problem making better decisions. Absolutely, no problem. That’s the great thing about this is that the skills already are in house. Cause we, we just have to apply the skills that we use to automate other people’s decision making. We just can’t apply them to our work. It’s pretty much like that.

Now the thing with legacy is, you’re right. Like legacy doesn’t necessarily have to mean very old. But in fact, it shouldn’t even mean something that is bad. Legacy should have a positive meaning, right? So in everywhere else, like legacy has this positive connotation. You want the legacy of … most of the time, right? But not in software. In software, people hear legacy, it was a woo. So, for example, another question I asked developers is, well, do you love working with legacy systems? And people definitely don’t associate legacy and love, you know, they don’t find a sentence that would make, make these two words work together in the same time, at the same time. So, and that’s a problem because legacy means value. It is the thing that produces value right now.

Now the other thing about systems is that right now we look at systems often as just transforming input into output. So this is one way, I call this one a utilitarian view on software systems. However, most software systems have replaced some sort of manual work that happened before. And we used to call that work knowledge work. So obviously, the systems, we can also look at them as encoded knowledge. When you look at the system as encoded knowledge, right? And then we start to talk about knowledge management, so that means exploiting the knowledge internalized in an organization to create a new value, then all of a sudden means it’s kind of important given that in any meaningful software organization, the core knowledge is not in people’s heads. And it’s not in documents, not in emails, or other forms of communication. They are in systems. Which means that any knowledge management based activity has to start from the core system. From inside the system. And that’s the opportunity.

And today, people look at legacy systems, and they see, well, this is an impediment. It drags me down. I can’t move, right? So this is why incumbents have difficulties when newcomers come into a space. Because they can’t move their systems while newcomers don’t have that problem until newcomers become large enough and then they have the problem with the next wave of newcomers. But if you would be able to change the system fast, then all of a sudden the legacy would become a competitive advantage, not an impediment. That’s where we want to be. So today, whenever people talk about legacy, they just talk about large costs, very high risks, and the difficulty of ever seeing inside. And I think we should be moving in a world where legacy is actually a positive thing. That we should be able to change it as well.

So this is one from an organizational point of view, first of all, just a single organization point of view. The idea would be take legacy from something that looks like a negative on the balance sheet to transform it into a value creation opportunity.

Let’s take an even larger perspective here. You know, let’s go to like societal level. So right now, we create software faster and faster. So year over year, the software grows super linearly. But at the same time, we’re unable to recycle all systems. So from this point of view, we behave no differently from the plastic industry. It’s not sustainable.

And so what does it mean to recycle the system? Recycle means to take it apart and refurbish it for new purposes. But of course, before you can take a system apart, you first have to understand the parts. But if the way I understand the parts depends on me reading, then I have a fundamental problem, because reading is capped at a constant speed. So on the one hand, I have a super linear growth, on the other hand, I have a constant recyclability function. It can’t work. It literally cannot work. So it’s not sustainable. So the only alternative here is to make it so that I can reason about my system. So I understand the parts regardless of how large the parts are and how large the system is. So the only way to achieve that is through tools.

But now, of course, we have these other problem is, of course, we want the tools. The problem is that software is contextual. So the only way to solve this problem is by making tools to also adapt to that context. That, again, leads us back to the same little tiny thing. So, you know, there’s little tiny detail of, oh, where, what do developers do?

Look how many implications it has. On the one hand, we talk about large costs associated with it. On the other hand, we talk about is this the opportunity for creating new value, like business, on the business side? Can I refurbish my system for that? But now if I just take an even larger perspective, I see, well, it’s actually not sustainable and we’re moving the whole economy on top of software, At least this new world should be sustainable, because we still have difficulties making the old world kind of work.

Henry Suryawirawan: I think very, very interesting perspective, right? So I think when we talk about legacy, we always associate it with bad things, you know, a lot of technical debt, very difficult to change. We just need to rewrite it, right? So that’s always the excuse of developers. We just need to rewrite it. But obviously, like what you mentioned, I fully agree. The challenge is actually to understand the encoded knowledge inside the code, inside the system, right? And it could span, you know, across multiple different places, right? And for me, in the past, when I join a new organization, a new team, they have existing system running, right, in the past.

What I try to do is basically to use instrumentation, you know, going to observability world. Yeah, you have telemetry, you have metrics. You have probably all these trace spans that you can understand how the system interact with each other. Sometimes if the tools can generate, you know, like automated diagram, that’s also one way how I did that. But it’s never fully enough, right? Because it’s always challenged. And sometimes if you have access to the data, you can also deduce it from the data. But we all know just by looking at data also not fully accurate because it’s just the output, right? You don’t know how it comes up, the calculations and all that. So definitely, this is a challenge. And I kind of laugh when you mentioned we are like a plastic industry. We keep churning software, but we cannot recycle them. I think it’s going to be a major challenge if we continue this path, right?

[00:30:17] Moldable Development Workflow

Henry Suryawirawan: So maybe let’s go into the moldable development, you know, maybe workflow, let’s say, right. So for people, because it’s still sound abstract for some of us. So how would you explain, you know, the workflow working with moldable development?

Tudor Girba: So let’s start with something you already mentioned. So let’s start with observability. The interesting thing with observability, of course, you can put the default metrics, you know, monitoring or so on, that level, CPU and disk usage. It tells you something, but not really interesting things, right? But then you can start to have, oh, let’s add some signals, which are maybe, technology specific all over this. I can start to learn new things. Like which services are being used. But then I can increase the specificity of it. Well, let me add the main specific, like business specific kind of signal so that I can, you know, answer business problems. Like how the business flows actually are representing my system. So then it becomes really exciting. But notice here how the value increases with the specificity of the signal. So that signal is a little tiny analysis.

There’s another place where we use little tiny analysis. This is testing. So in testing, you have a problem, and then you stop, you write a test. And in the test, you say, well, this is the setup and this is what the condition should be. This condition captures value. This assertion here says this is valuable. And the consequence is that as soon as a single test fails, people stop and fix it in most of the time today, right? So you’re not going to deploy if the tests are not green. So that’s pretty nice because you’re stopping the line to deploy even if a single test fails.

Now if you switch to, let’s say for example, a static analysis. Again, static analysis, like a test, is nothing but an analysis tool, right? It takes functionality and transforms it into red and green types of things. So it’s kind of interesting, right? But static analysis is also an analysis. It takes something, in this case, it takes static source code. It also produces a red and green type of thing if I look in the range of rule or something. But then I can have systems where I have, you know, thousands of tests or tens of thousands of tests, all green. And then I have on these other places, I have tens of thousands of warnings. How come? Is it like this team all of a sudden, you know, lost interest in qualities? Like what happened? Like do they have two minds? Why is it?

Well, the thing is just look at the difference in how these two analyses have been produced. So the tests have been created within the context of the system after you know the problem. The static analysis typically is downloaded from somewhere. So you’re downloading it from your system for your system, and I don’t know, you’re doing, I don’t know, maybe banking or so. And I’m working in embedded systems. And now all of a sudden, we’re using the same analysis. You use the same, I use the same. So by definition, that analysis must capture what’s common between us, which also by definition, it doesn’t capture what’s interesting about our systems. Of course, you don’t want to mess up null, you know, send, create null pointer exceptions. Of course, you don’t want to do that. But that’s such a terribly low value, right? Like you’re leaving the rest of 99.99% of value creation, right, is left completely unchecked.

So these things, these static analysis that we download from the web, they solve a problem, but they literally solve someone else’s problem. So the alternative is we can take the same thing that works for testing, the same thing that works for observability, but literally apply it to everything, including the tools that we look through. So not only checks of sorts, but also the tools that we visualize, view things, or edit things.

And this is also kind of interesting, because it leads us into the development experience space. At the very beginning, you mentioned the IDE. I in IDE stands for integrated, which basically means every time you leave the environment, the I has failed. And today, like, people use things that are called IDEs and they spend like 70% of the time in the web browser. This is really funny.

And then the thing is, right, the question is, well, how do you make this development experience work? And one thing to realize here is that you cannot experience anything in a system except through some sort of a tool. There’s no way for me to sense anything in a system, maybe if I’m looking at the code, maybe I see an editor. Or if I’m looking at an object, maybe I was going to see any object inspector or something like that. So there’s always a tool in between. But the interesting thing is, I mean, what’s something we know from data science now is that if you just change the way you look, you don’t change the data, you just change the way you look at it, and you can fundamentally affect the way people make decisions about it.

And this ability, the ability to control the thing we look through. And this also includes the signals in observability, the static checks, but also like visualizations, queries against the code. And not only against the code, but against what if my system is built now in multiple languages and it has a service oriented architecture. And maybe there’s also a bus in the middle, there are messages being sent there. And if you look at how those are captured in the code, there’s no dependency between this. You know, the message name is going to be some string somewhere, and that’s going to be pushed. And then somebody are going to read that string, and it’s going to say, yeah, you just sent me a message, which means that there’s a dependency there, but we don’t see it. If I just look at the base language, I don’t see anything there, right? But if on the other hand I can, if I elevate the tool to understand my framework, for example, the framework that I’m using, all of a sudden, I can start to see things that were completely invisible before.

During the pandemic, for example, we donated our time for a government and we build an app. So we helped build an app, but we used React Native. But we didn’t. We didn’t know React native beforehand, so we had to learn and we had one month to produce the app. So we spent, I don’t know, a few days to learn React Native, so we did. And we started to build components. It was not a complicated app. Maybe the second week or so, we ended up with the dependency between like the cyclic dependency between React components. And we said, okay, well, so React is only the most used framework in the world. So obviously, we think maybe others have that problem. And surely, surely enough right, they do. So you look into Stake Overflows of the world, like there’s lots of those kinds of problems. How do I deal with the cycles? But at the time, we said well, okay. So again, this is a very widely used framework. It looks to be a rather often recurring problem. Surely, there must be some tools that will help us detect these cycles. We didn’t find any.

So, the people have the time. And just think about this Stack Overflow, right? The proposition of Stack Overflow is I have a problem in my system. The thing that I chose to put together tells me here’s the error. And the best possible engineering solution for me is to take this problem, go and ask it somewhere else and hope that someone that has never seen my system will give me the interesting answer. This is absolutely not engineering, right? So to me, the success of Stack Overflow, and Stack Overflow was asking, until a couple of years ago this interesting question, which was, what’s the first thing you do when you have a problem?

And there was always, I think there were a couple of years where there was, you know, do you Google first or do you go to Stack Overflow directly? So which one is it? It was not clear. And it was like 90% people responding, this is the first thing I do when I have a problem. So that’s literally like there’s a huge chorus of people really crying for help. Because that’s not an engineering way, right? We shouldn’t, this should be like the last resort. Sometimes maybe you want to go and debate and so on. But this should not be the first thing we think about.

So when we had that problem, what did we do? Well, we stopped and we built a tool that would show us here the components in React and we were being able to click on the component and see, here, where it is. But it is so even this simple thing, right?

So if you think about your development, your editor, right? So what I like about Visual Studio Code is how honest it is, right? It doesn’t sell itself as an IDE. It just says I’m an editor, which is really, really nice and honest. So in an editor, you typically see files. And the source code. I just described, like in the React application, I really want to see components. I don’t want to see files. Files are very low level. This is the, like the lowest level in which I’m storing data. Why would I want to look in the database and just see a record by record? Like why would that be an interesting view of things, right?

So what I really would like to see, I would like to see an index of components. It’s a very basic thing I should have an expectation for. But because we don’t talk about it, we don’t expect it. So I don’t like to do that. So when I have a problem with components, I want to see the list of components. Okay, here’s a list of components. If I can see the list of components, and if I can say, well, this component depends on the other component, because it’s the other crazy thing. If you think about, oh, JavaScript, it’s very weakly typed. It’s hard to do static analysis. That’s kind of true. But if you look at React, that’s hard coded type. It literally says this component, it depends on these other components. It’s just like the names are right there. There is no lookup, there is nothing. It’s very, very concrete. So you can actually build very cheaply a tool that really gives you the graph of components. It’s not difficult.

But this mass hallucination space that we live in where we think, oh, I don’t know what to do, I got to go and ask on Stack Overflow. You don’t do that. Stop and look around and look at the opportunities that are around you. And you will see actually going to Stack Overflow is the first thing. Oh, okay. Now I’m not going to go to Stack Overflow. I’m going to just type in, because ChatGPT is much closer. So I’m going to ask, oh, what are those, right? And ChatGPT will tell me, oh, here are the components. But you have no way of knowing if these are all components or not. Maybe there are, maybe they’re not.

But there is a way to use, for example, something like ChatGPT. We’ll say, well, what if you asked your GPT, could you give me a tool that I can run, a small tool that I can run to list all the components? And maybe if I say, oh, look, actually, you missed this one over here, because you know, like, there are not just one way of defining components, for example. Nowadays in React, we can have several ways. So now if I find this exception here, maybe I go extend my tool. Maybe with AI, maybe without AI. But the idea of having an unmediating, inspectable, deterministic way to answer questions about the reality of my current system is really simple and is incredibly powerful.

And that’s the essence of moldable development. We’re simply saying, now imagine you have the possibility to create custom tools for every problem you have. That’s the essence of moldable development. Now this, of course, means those tools have to be creatable at much lower cost than we consider tools to be created today. So when I say much lower cost, I mean minutes cheap. So within minutes, I should be able to create an increment that gives me some sort of added value. Maybe it’s not the end of it, but if I spend minutes, I should, I want to see something that produced an increment and I can decide whether or not I go left or right. So now this means that I now need a whole new kind of a development environment that makes it whose first job is to make the creation easier. Because anything that comes out of the box solves some problem, but it’s almost guaranteed to not be my problem. So this is more interesting for a development environment to start asking, not how do you edit this, but how do I create a new tool that is highly specific to what I want?

And so this then leads to a new kind of a development environment, but also implies new kinds of skills. Because now I would have to have the skill of creating the tool ideally within the organization, within the team. But even more interesting is the skill of asking questions. So today, because we answer questions very slowly, we’re going to ask very few questions. Interesting questions about the system. And because we ask few questions and far apart, the muscle, that is the muscle of asking questions, atrophies. So we find ourselves asking kind of generic questions. And they’re very difficult to act upon and very difficult to answer. While what we want to do is we want, like if you think about, okay, here’s the question, here’s the answer, and then here’s the next question that comes.

So, if the distance is very far between the question and the answer, the distance between two questions is going to be very far. But now imagine then I can decrease the cost of an answer to be almost zero. Not zero, but very, very close, many times faster. All of a sudden the bottleneck will be, oh, how do I come up with the next interesting question and the next interesting question and the next interesting question? And so the skill of learning how to build tools. So of course, you need the environment, you need something where you can create tools very quickly. And when I say create tools, I mean thousands of tools. When I say thousands of tools, I mean thousands of tools per system. So little tiny micro tools that coexist and I can use to assemble narratives about my system very quickly and pretty much on demand.

So if you just think about this, so in my world, I’ve seen, for example, a large organization like a large unicorn and they had thousands of developers. And then when we looked at the tools, they use dozens of tools, right? Now, okay, in my organization, my team, we are a dozen people and we use tens of thousands of tools. And so it’s similar, right? Everybody uses tools, but not the same way. It’s kind of like when people were saying, oh, we do some testing. You know, we do some automatic testing. So for these five cases here, this is automated, but the rest of the 10,000 scenarios are not, right. Or when we’re saying, oh, some parts of the deployment pipeline was automated.

Yeah, so there’s like some automation and it looks like deployment, but it’s not the same as when you have a deployment pipeline that you can use 10 times per day. It’s like the properties are not the same. So that’s why we’re saying and so you need this technology to allow you to create and host and effectively use such tools. Maintain them. You need the skill of creating the tool. But the most interesting and important skill is going to be the one that is about asking questions.

Henry Suryawirawan: Yeah. In your article with Simon Wardley, you mentioned about time to question, time to answer, right? So I think that’s very critical. Maybe the critical thinking skills that people are talking about lately, you know, with the advent of AI, right? We are losing our critical thinking. So maybe asking questions and getting the answers is definitely still going to be the main core skills for developers.

And actually, when you mentioned about cyclic dependency, I kind of like remember my recent experience, actually. I was working with a NestJS backend system, right? And we have the concept of modules and service and all that, which also has a cyclic dependency kind of problem. But I could actually solve the problem just by subscribing to their tools, which they call DevTools, $5 a month, just to visualize how module interconnect with each other. So actually, I kind of understand what you’re saying, right? So imagine if you have all your files, all your classes and all that, but you can visualize it differently. Maybe through different type of micro tools that you mentioned, right? Maybe you could even visualize and solve the problem faster. And best case is, like, you start generating more questions, like how can I improve the system, or maybe there are other parts of systems that I would like also to know about, right? So I think this is definitely very exciting.

[00:46:02] Glamorous Toolkit

Henry Suryawirawan: And you come up with this tool called Glamorous Toolkit, right? So maybe a little bit to share with people how can they use Glamorous Toolkit, what kind of things that they need to do in order to come up with these micro tools?

Tudor Girba: It’s interesting that you mentioned, right? So you have a framework, and apparently, people that build the framework came up with the idea that they should also build tools. They don’t understand the framework. I mean, this is such a, it should be so common knowledge, right? But the thing is, even when you don’t have the $5 per month or day or whatever possibility. It shouldn’t mean that you’re hopeless. It should still be within the realm of possibility to literally build your own tools. So we as a tiny, tiny team, like we were like four people, so four people decided, oh, you know what? Like we see this very large problem. Everybody has it, but it’s economically feasible for us to spend a day and build a tool.

And then we have a tool that shows me the dependencies and cycles between my components. And then the thing is that, you know, tomorrow you will not be using a very commonly used tool. And maybe you will not just be using one technology, but you’re using a combination of things. And all of a sudden, then what? There is no off the shelf, there is no $5 per month type of opportunity. So you, what do you do? Oh, you’re going to go to Stack Overflow. So there should be a space, a continuum, where I say, well, what if for any question I have, there is an economic way, an economically feasible way to go and answer questions.

And so that’s what Glamorous Toolkit is. Glamorous Toolkit is a free and open source environment that we have created. I mean, this is a really crazy idea if you look at the status quo, right? Like what I’m saying here, thousands of tools, like how do you maintain this? People say, oh, are you not going to be bombarded by too many tools? And the answer is yes, you can maintain it very quickly or easily and cheaply. Actually, it’s much cheaper than if you don’t build the tools. And yes, you’re not going bombarded, you’re not overwhelmed by thousands of tools because you don’t see thousands of at once. You see them only those that are meaningful in a specific context. So it was a really crazy idea a decade ago.

And so the current incarnation of Glamorous Toolkit, we started about 2017. And the goal here was we want to create a vehicle with which we’re going to explore how far this idea can go. And so, the way we set up was that if we are going to exist in 10 years, it means that moldable development works. So, you know, when you’re doing your own validation for a new idea, you’re going to be biased, right? So it’s obvious if I say, oh, look, I’ve done this one. I applied in a case. It works brilliantly. Everybody should do exactly this, right? So that’s, we see that it economically works in all sorts of situations. But not often necessarily, you know, it’s not necessarily grounded in reality. So we really wanted to validate before we’re going to go.

So feenk was created 10 years ago and we still exist. And the way we created it was we said, well, okay, we can’t do unbiased type of validation, because we’ll have to do the validation ourselves. So what are we going to do is we’re going to create the most adversarial type of situation we can imagine, and if we can survive that, then the only way, the only explanation would be that this moldable development works. So Glamourous Toolkit is among those. Everything we do is free and open source. Glamourous Toolkit is a significant piece of technology. It’s used in production in different places. We made it completely free and there is no, we don’t have any means to charge it. So people can just download it and work with it.

But when I say it’s significant, I also say, well, we’ve actually used it in dozens of different classes of problems. So not problems, but like classes of problems. So when I say, you know, like classes. I mean, how do I reverse engineer an API? How do I reverse engineer a protocol? How do I make a distributed system and reverse engineer and move it to something else? How do I reverse engineer architectures? How do I reason about the domain and make business answer like do explainable domain driven design directly from the system? How do I do data lineage? How do I do DevOps and observability directly from there? And all using the same kinds of tools and techniques.

And what they, what we are showing, and when you open Glamorous Toolkit, it comes with all, like a book. A book of the size of Harry Potter written inside in the environment, because it’s a knowledge base. It’s a data science platform. It’s a programming tool. But this is not like Atlassian does it by just stitching together different tools, but just having a really common design. Because these are not different spaces. And what we are showing is that it’s possible to tackle a really, really different classes of problems exactly in the same way. And the promise here is if you can do that, it means that you can invest in the skill once and then use it over and over again in all sorts of unforeseen types of situations.

Now when we build Glamorous Toolkit, so we used it as a vehicle for us to explore. But at the same time, Glamorous Toolkit is the first large case study of moldable development. If you look at something like IDEA IntelliJ, and you look into the plugins marketplace, you will have something like 6-7,000 plugins worldwide. Then you move to Visual Studio Code. And I think last time I looked at what they had, like 60,000 plugins worldwide. When you open Glamourous Toolkit, built by a tiny team, you have more than 4,000 plugins in the core. And these are the tools that we used to develop the platform itself. So when people are going to use, take Glamourous Toolkit and use it for another system, they’re going to be building thousands of tools more per system, not worldwide. Per system.

So Glamorous Toolkit is the first major or a significant case study of moldable development itself, which means that you can use it and you can take it and use it for solving actual problems. But you can also just take it and use it and say, well, what does my moldable development mean? And simply learn from it, and then maybe copy it in other places. Our only worry is that people will not copy the whole of it. Because this thing has so many implications, so many interesting details, and it can be so powerful. It’s not just about your view things, but also how you create editors for different languages, how do you combine different languages, how do you deal with data in different formats, and so on.

So on the practical side, the way you work with Glamorous Toolkit is, it’s based on Smalltalk. So it’s an image based system. So you basically download it and then you customize things. Sources are all on GitHub. So you can also create your own customizations putting next to your project. So you can take and build like a docker, you can take a base docker, create, put something into it. So take the base image, put something into it and save it. And this is going to be the distribution for your system. So this is how it is.

The way you learn it, we advise people to split learning into phases. On the one hand learn how to learn. And then afterwards, learn for a specific problem. And this learning how to learn is really interesting. Because as I said, GT, Glamourous Toolkit, was built following moldable development so it has explanatory tools inside, so it has a way to search, for example. Inside has the way to understand like different objects will have different ways to visualize it themselves. So learning how to jump from one tool to another is kind of important.

And the other thing here is if you think about tools that you typically think about features. So you might be learning things like shortcuts by hand or by heart, um, or you’ll say, well, this button does this, this button, then that, right? There’ll be dozens of buttons to learn, or maybe like a hundred. Okay. But in our case, we have thousands. And then when I start to combine them, I have an explosion of possible combinations of things. So it’s not possible to learn them by heart. Instead, what you want is you want to learn what’s this? How do you decompose each problem in the same way? That’s the most interesting thing. And this is how we can scale to arbitrary situations and actually not overwhelm people.

Henry Suryawirawan: Sounds really interesting tool, right? So definitely we’ll put it in the show notes for people who are interested when you listen Tudor’s explanations just now. Sounds really cool, actually, if you can actually come up with different ways to answer questions about your systems, right?

[00:54:15] IDE, AI, and Glamorous Toolkit

Henry Suryawirawan: And speaking about tools these days, I’m sure people use IDE, right? Not just simple text editor. And people now have AI tools as well, either chat based or maybe the Cursor or whatever that is, right? It’s more like agentic AI. And I’m sure this Glamorous Toolkit is one potential tool, although the traction probably we still need to see in the next few years. Where do you see the future of our development environment, right? Do you actually see IDE, AI, and Glamorous Toolkit living side by side, collaborate, or even can we use AI to come up with those micro tools?

Tudor Girba: Well, so first of all, Glamorous Toolkit is a development environment, so you should see it. It’s just that it doesn’t look like any development environment. Because typically we open a development environment and what do you see? A giant text editor in front of you. What are you going to do, right? You’re going to scroll through it and try to learn, right? There’s not even a search box. So in our case, right, because anything is the last thing I want to do. Not the first thing I want to do. Just think of this. What does it mean when I say, I said at the beginning, we don’t talk about how we read code. Where do people read code? Well, in an editor. It literally wasn’t made for reading. It was made for editing. It’s a different use case. It’s similar but it’s really different. So the question is why should you see an editor as the first thing? I don’t know why.

So we used to have, for example, Yahoo was having, oh, here’s the tree of all the interesting pages. And then there’s other guys will come and say, well, what if I just give you a search box? Much more interesting as an interface, right? And then nobody would say, well, where is your tree? But I still remember, actually, people asking that question, where is the tree? Well, you might not be interested in the tree, in fact.

So yeah. So I think that, that if you optimize, if you take seriously this idea that we should be optimizing for decision making, the nature of the tools should change. They shouldn’t be, it’s not an increment. It’s a really, really different kind of perspective. So Glamorous Toolkit is a development environment, but it’s a development environment but you can create other development environments.

Now I do believe AI has, like generative AI as the way we see it today, it’s very interesting. Not really in the way necessarily people look at it today. So I really think that as long as we want to be the ones making choices about what goes into production, we have to solve the reading problem. There is no way around it. Because otherwise, we’re just going to pretend. Which is exactly what we’re doing right now when people pretend to work and then three years go past and nothing changed. But then at some point some people will exploit this new opportunity of using our, leveraging the human intelligence. And they will transform this one into a competitive advantage, and this will become the new norm.

Just like how DevOps, right? So even today, we still find people that are still now automating their deployment pipeline. But they will be automating right after they’ll put automating testing in place. There’s going to be a long tail of those cases. And it’s the same thing I believe will happen with the way we optimize decisions.

So what AI is is very interesting. So if you use it for generating arbitrary kinds of code as people are trying to do, now I think that this, the appeal there is right. So the local platforms are in trouble or they have to become AI. But local never really displaced the interesting systems. As long as we as humans decide we want to understand what happens inside the system, we have to solve the human problem, the reading problem. There is no way around this. Because literally today, we humans or engineers and business people, are starved for information. People don’t see the system. You literally don’t have, like your brain can’t work, it doesn’t matter at which speed, because you don’t have the input. So you don’t know what’s, what’s in there.

So when you’re asking the AI, right, the other use that people use AI for is, oh, we’re going to use AI to summarize the system for us. The problem with that is that again, you don’t know what of it, what of the answer was left out and how covering is this one. So basically you’re replacing someone’s drawing on the whiteboard with this one. But there are beliefs in both of them, but they’re not quantifiable. And so that is not an interesting way, I think. It’s very appealing, but it’s not really an interesting engineering solution. On the other hand, if you ask an AI to say, well, could you create me the tool that I can run to see these dependencies, for example. That I believe is very interesting because those tools can be really small.

But then, of course, now all of a sudden I want to create lots of those tools. Why shouldn’t I be creating lots of those tools for any of my questions? Then all of a sudden you want to demand, oh, I need to put somewhere these tools. So I will be demanding that my development environment should be hosting these kinds of tools. So maybe all of a sudden, this editor, just giant text editor that I have in front of me, is not the most interesting thing at all to see.

So I’ll be demanding, you know what, I should be able to have a different kind of a development environment there that allows me to have an engineering conversation with my system. And then once you see this and say, well actually, you know what, if the cost of a tool is actually so tiny that the blocker isn’t the creation of the tool, but is the asking of the question. Then all of a sudden, actually, the AI is just, it’s interesting optimizations, but it’s nothing but an optimization.

So that’s what I see, because at the end of the day, it doesn’t matter who creates the code. It matters who has to make a choice about that code. So either we’re going to delegate everything to an AI, and I can see lots of incentives for doing this, just like how people, it’s kind of trading which is completely unmediated by humans. Today, algorithmic training. And I can see this one being extrapolated, and there will be classes of problems where people will want to do that. Like, for example, spam. I think they are thriving today. There are classes of problems where this would work. But there will be classes of problems where I don’t see this current technology ever doing this. Other kinds of advances in technology might change the equation. But the current one, there will be interesting, very large areas of economic value creation, where the human will have to decide. And then that is the problem that we actually have to solve.

Henry Suryawirawan: Right. So I think that’s really insightful explanation, right? So how probably we can see a potential future where maybe AI, Glamorous Toolkit, moldable development, right, working side by side. And we solve different class of problem.

[01:00:36] Writing with Simon Wardley

Henry Suryawirawan: So I noticed also, recently, you kind of like co-wrote a book with Simon Wardley. You also mentioned it in very beginning. Maybe can you tell us a little bit, what’s the mission here? Like, why are you writing the book? What are you trying to come up with Simon, right? And what are you trying to explain to people?

Tudor Girba: Yeah. So first just to wrap up on the AI and Glamorous Toolkit. In Glamorous Toolkit, for example, we think AI is really interesting. But the way you’re going to consume the output of the AI is really important. So we’re actually doing research. And there is an infrastructure in Glamorous Toolkit directly to work with AI, but create on demand interfaces to consume the output of the AI. So, for example, imagine starting a chat. And just having a generic chat interface. And then while chatting to customize how the messages are being proposed to you. So if you’re seeing something, by default these days people say, oh, if I show you a picture is like this. What if I want to see the hex of the picture, right? Or if the encoding or something like this. Or so I might want to choose different kinds of representations. And so that interface, controlling the tool through which you look, has lots and lots of implications.

This is a bit, maybe what I’m saying here is a little abstract, hard to believe. I understand that, and that’s the reason why we’re writing this book, Simon and I. We worked for several years together. And since about a year, he chose to work much more closer with us. We basically spent the whole year trying to find ways to explain what it is that we discovered in our research and in our work in a way that is more consumable. So something that abstracts a bit the technical details.

And find a way to explain to, for example, a CTO or CIO, maybe CFO type of level as to why is it interesting to look into the legacy space? How do you create value? How can you create value out of it? And why the development experience is the answer at all? And of course, to explain on moldable development. That’s a little bit of the background.

Henry Suryawirawan: So definitely good luck with the rest of the chapters that you’re writing with Simon. I read all the chapters that you have so far, right? So I must say it’s very interesting. It gives a lot of background and kind of like the problem challenges we are facing. And I’m sure we got to learn a lot more about moldable development in the future.

[01:03:01] 1 Tech Lead Wisdom

Henry Suryawirawan: Tudor, it’s been a pleasure to have you in the show. I have one last question, which I normally ask to all my guests. I call this the three technical leadership wisdom. Think of them just like an advice you want to give to us. Maybe if you can share your version of three technical leadership wisdom.

Tudor Girba: I think I’ll just stick with one, one of them, which is simply start to spend 15 minutes a day as a developer looking for how else would you be able to find an answer to a problem that you’re looking at right now? Don’t try to necessarily do it. Simply just think about it. And then what you will see is that after a couple of days, you will start to notice opportunities popping up around you. Because the interesting thing about whenever something is not explicit, your eyes are not trained to see interesting things related to that. And I think this is an interesting problem in any case, because it goes into this idea that whatever you’re training, that’s what you’re going to use.

And teaching yourself going through a more inconvenient experience as like everybody else is doing something. I’m just telling you now, right? Go ahead and not do what everybody else is doing. Try to think, how would I possibly answer this question here but without reading a line of code, if possible. And sometimes it’s not exactly possible, but many times it is. You might find it around you. And so this exercise, I find is an interesting exercise especially with newcomers in programming. Because it teaches people to start to train the ability to see things that are not obvious at first. And the other thing that it trains is the idea that engineering is a discipline that doesn’t follow hypes. Hypes might think, oh, engineering changes all the time, but it really doesn’t, right? And there can be a systematic way to discover it. And so that’s what I would urge you to do it.

Henry Suryawirawan: So think how you would think, right? So it’s kind of like meta level. So I think it’s kind of like beautiful advice, right? So I think we could use more critical thinking. And also kind of like probe the system that we have, right? So that we can understand it better. And maybe there are opportunities, like what you said, how we can improve it so that our software can be recycled, not like the problem that we are having now.

So Tudor, if people would love to connect with you, maybe they want to ask you more questions. Is there a place where they can find you online?

Tudor Girba: Well, yeah, so of course, you can find me on LinkedIn. I believe at Girba on LinkedIn. And I invite people to just connect with me and directly ask questions. You can go to Glamorous Toolkit. You can download at gtoolkit.com. There’s also a Discord community there. And if you’re interested in what we do as a company and how we modernize legacy systems, go to feenk.com. So F E E N K, comes from feel and think. Yeah, find more details about it there.

Henry Suryawirawan: I like it. Feel and think. feenk.com, right? So, if people have modernization challenges, you can contact Tudor.

Tudor Girba: Just wanted to say that the feel and think comes from the fact that we know how to think in our world, right? Or we think we do. But we forgot that the way we feel is as important as the way we think, right? And if you come back to this problem of how do people feel when they are in the presence of legacy? I believe we should change that.

Henry Suryawirawan: Really nice name behind it, right. So I think, for people who are interested in moldable development, make sure to check out Tudor’s resources. And I hope you have more traction in this space, right. So looking forward to see the future where we are coming up with our micro tools.

– End –