#93 - Maximum Value Maximum Speed Software - Dave Thomas

[00:01:24] Episode Introduction

Henry Suryawirawan: Hello to all of you, my friends and my listeners. Welcome to the Tech Lead Journal podcast, the show where you can learn about technical leadership and excellence from my conversations with great thought leaders out there. And this is the episode number 93. Thanks for tuning in and listening to this episode. If this is your first time listening to Tech Lead Journal, don’t forget to subscribe and follow the show on your podcast app, and social media on LinkedIn, Twitter, and Instagram. And for those of you who enjoy this podcast and wanting to contribute to the creation of the future episodes, support me by subscribing as a patron at techleadjournal.dev/patron.

My guest for today’s episode is Dave Thomas. Dave is the chairman of the Bedarra Corp, (founder of OTI creator or IBM Smalltalk and Eclipse,) founder of YOW! Australia and Lambda Jam conferences, and a GOTO Conference Fellow. In this episode, Dave shared about his personal research, which he currently refers as 42D, on ideas that we can use to develop high value software rapidly. He started by describing the current developer’s productivity challenges, and touched on the idea that big is not better, relating to the size of the team and code base, and how he finds the development tools are becoming more and more complicated and complex. We then discussed on the importance of developers understanding domain knowledge, leveraging tools such as decision tables and spreadsheets. And we also discussed how the choice of programming language actually matters. Towards the end, Dave shared about using a data centric approach to deal and build upon legacy systems. Towards the end, Dave shared about using a data centric approach to deal and build upon legacy systems, and also shared his perspective on query as the future of programming.

I enjoyed my conversation with Dave. As always, he shares very insightful perspectives, especially taken from his vast experience. And I’m always amazed by the different exotic programming languages that he talks about. If you also enjoy this episode, please share it with your friends and colleagues who can also benefit from listening to this episode. Leave a rating and review on your podcast app and share your comments or feedback about this episode on social media. It is my ultimate mission to make this podcast available to more people, and I need your help to support me towards fulfilling my mission. Before we continue to the conversation, let’s hear some words from our sponsor.

[00:05:11] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today I have someone with me who I always admired whenever he gave presentations when he came by to Singapore. He’s also the founder of YOW! Australia, and also the GOTO Conference Fellow. His name is Dave Thomas. I always find your talks very insightful. So really looking forward to this conversation, Dave, and I’m very pleased to have this chance to talk to you today.

Dave Thomas: Henry, thanks very much. You have a great show. Really enjoy listening to it and catching them online. Yes, there are two Dave Thomases. Fortunately, he’s a very good friend. He was doing Ruby, and prior to that, I was doing Smalltalk, which you could sort of think of as the first Ruby, if you want. Dave’s now doing Erlang and JavaScript. I’m dabbling in vector functional programming is where I’ve been for the last little while. Great to be here.

[00:06:00] Career Journey

Henry Suryawirawan: So maybe for people who may not know you yet, if you can introduce yourself, spending a few minutes to tell us your journey. Telling us more about your highlights or any turning points in your career.

Dave Thomas: Henry, you know, you have to be careful asking a software fossil about their history. I could put the entire audience to sleep. I’ve been really fortunate to work with a lot of amazing people. I think that’s one of the advantages of being early in the field is there weren’t very many books and there weren’t that many experts. So I think in many ways it was a lot easier to become an expert because you didn’t actually have to know that much. So I was very fortunate to be there. I was able to work my way through college paying by programming. Helping the students in business do their really tedious assignments so they could go skiing. They would pay good money for that. So, I was fortunate to work in a number of areas during work terms, and that was really great.

And then after graduation, I was lucky to get a job in the (university) computer center. Computer center probably would not be the place that maybe some people would go look today, but in the early days, I mean, that’s where Simula 67 was invented. It was the Norwegian computer center. It was really the place if you wanted to see programming languages and operating systems. That was really the only place you could see it. You could get in that big computer room with all that air conditioning and be there with the machine. So it was really quite exciting. So I ran the programming languages group there and worked with both the Honeywell and Xerox who were computer companies in those days. We were able to, through our relationships, helping them, end up getting contracts, develop the Pascal compiler and compiler tools and database refactoring technologies. So we’ve developed quite a good relationship, and was fortunate I got to see a lot of the US industry from the inside, and how they manage software projects, or didn’t manage software projects as the case may be.

I went back to grad school. Did graduate work and specialized in databases and programming languages. And just as I was finishing grad school, the Dean of the school of business came and he said, “You’ve been teaching our students for some time. We really have a terrible old-fashioned IT program. Would you come and be a faculty member and redesign the program? So I was so privileged to do that. I did a lot of teaching. It’s a privilege to teach. Students keep you honest. They ask those naive, really hard questions, which are always great to have to think about. I started there and soon after that, the Carleton University here in Ottawa, Canada decided that they wanted to have a new School of Computer Science. And so I was invited to participate in the founding of that school. I ended up with a cross appointment between business and computer science. My undergrad is in electrical engineering. But that’s the nice thing about computing. You get to go to different disciplines, work with people in biology and finance and so on.

I was really interested in office automation. I came across this work called the system for business automation. That took me down a very interesting path into languages like Scheme and particularly Carl Hewitt’s Actor (model), which is kind of the inspiration of things like Akka and so on, the route there commercially. That led me fairly quickly towards this land called Smalltalk and object orientation. So I was really interested in that. We actually built an implementation of Smalltalk from the Byte magazine in 1980, implemented in a very exotic language called APL. There’ll be many cruel and unnatural acts on my journey. I love exotic languages.

So, a friend of mine called me up who was in my engineering class and said, “Look, I’m building this hardware company, but we don’t know anything about software. You’ve got to come and work with me.” So I took my sabbatical working in a company that built Z80 and MC68000 micro computers. We built a network operating system, essentially like a network file appliance. We attended the first course by Brad Cox, who was a GTE then. He had a language called OOPC, which you would only know today as Objective C. So that was the beginning. It wasn’t ready. So we built our own language and compiled it to Forth. Forth is a very interesting language. Many of the tricks we used there, we actually used later in Smalltalk and Java for building embedded systems. How to put things in ROM and flash and deal with garbage collection in really complex systems.

So after I worked at this company Dy4, I went back to Uni and I was really convinced that I needed to work on the real thing. I needed to really find out about objects. The only way to do that was really to study Smalltalk. Fortunately, we were able to get early Smalltalk from Berkeley, Dave Unger’s work on Soar Digitalk Methods and from Tektronix, who at the time were building AI workstations. We were fortunate to get our research funded by the DND in Canada, who were looking for a way to build models of multiprocessor systems. So they funded us to build Actra, which is a multiprocessor Smalltalk system. They also funded ENVY/Developer, which is the easiest way to describe is sort of a Git for Smalltalk done circa 1986. That allowed Smalltalk programmers who normally work in an image, and you do not cooperate to actually share and do version management, fine grain the level, and that was our first product. Then both Tektronix and DND funded us to build a production Smalltalk, and the Smalltalk we built for Tektronix, embedded Smalltalk V, was actually put into the Tektronix oscilloscopes. And those oscilloscopes started shipping in 1988. Before Java, there were bytecodes inside of embedded systems. Several of them including later on the Momenta pen computer and a first version of the Sony Qualcomm mobile phone prototypes in the early nineties. So that was really exciting. Smalltalk was a wonderful time.

In 1997, OTI, which was the company that did this work, was purchased by IBM. Of course, before that had built IBM Smalltalk, which was the basis for the IBM VisualAge platform. After we were purchased, we were given the mission to develop VisualAge for Java and the IBM JVMs, known as J9 still to this day. We were handed the enjoyable and challenging opportunity to develop that really without access to the source code from Sun because we would be contaminated. So, a lot of the core libraries and so on were actually written from scratch for the J9 virtual machine and IDE. At the time, we started with the Eclipse project. Eclipse had another part called the UVM, universal virtual machine, which was a multilanguage virtual machine. But unfortunately, IBM decided that they were only going to go ahead with Java. At that point, Eclipse was off and running, but it became only a Java IDE. Originally it was designed to be a multilanguage IDE. That’s the way the journey goes. I left IBM about that time.

At that point, everyone was getting quite excited about something called XP and Scrum. Fortunately, we had a process at OTI called Just-in-time software, which was really very much in the spirit of XP. I joined up with a bunch of other enthusiasts to form something called the Agile Alliance, which I sort of apologized because Agile has been such a depressing thing the way it’s been implemented. It seemed like such a good idea. How could something so simple go so wrong? Oh, well. My apology. Objects do have their problems too. They’re really powerful, but they’re very stateful and you can make things complicated. Learned some lessons from these things.

After that, I did a lot of training and transitioning the people to OO. And then, I had to withdraw myself and go back to a place called Bedarra Research Labs, with my former partner from OTI, Brian Berry. Focused on research. Our research themes were collaborative analytics, accelerated development, and embedded systems. We did one project in embedded systems. And then a Canadian government agency, CSE, came along, and said, “We have this big problem with cyber analytics. We’re using these really old tools. We can’t keep up with the data volumes and the speed.” So they said, “We’ll fund you to try your next crazy thing.” They’d already used Smalltalk and Eclipse I had done in the past. We’ll try whatever you’re doing this time. Very fortunate to have these kinds of customers.

So we built something we called Ivy, which is, essentially, a visual workbench for collaborative analytics. We sold that to a company called First Derivatives. We created Kx Labs. I was the chief scientist there until a couple of years ago.

In the last 18 months, what I’ve been doing is, fortunately, through Zoom, interacting with a bunch of interesting people around the world, which is I think the good news about the pandemic, I’ve started my own personal research journey now that I have time to go back and look at some of the things that I found very interesting, but didn’t have time to polish, really understand what was good or bad about them. I’m now able to do that at leisurely pace. I’m sure it’ll keep me busy for the remaining time than I’m able to use my technical brain.

Henry Suryawirawan: Thank you for sharing such a comprehensive journey of yours. So every time I listened to your story, you did this in your talks as well, right? So I’m always fascinated. I didn’t know many of those names that you’ve mentioned. Maybe also because due to my age. But always excited to hear, okay, what are these things that Dave is talking about? It seems like very exotic, very fascinating. And then your story also told a lot of thin lines across programming languages, object-oriented, Agile, data.

[00:15:10] 42D

Henry Suryawirawan: Today we are going to talk maybe a personal project of yours as well. As part of your personal research, which is called 42D. Maybe the name sounds interesting as well, quite exotic. Can you tell us a little bit more what is this 42D project?

Dave Thomas: Sure. 42D is really just a personal project. 42, of course, everyone should know is the answer to life and the meaning of the universe. So 42D is just one small galaxy, and it’s the galaxy where we’re seeking to explore and find ways so that small groups of developers can develop high-value software quickly. That’s really been my interest every time we start a new product, a new application. Is there some way we can change the process, change the tooling, change how we think, such that we can be more productive and not end up with big, ugly code mountain that we always seem to end up with. That’s what 42D is about, and it’s really just taking a few things that I’ve found to be really useful over time. Trying to now think about them and see if I can bring them to be more mature ideas.

Henry Suryawirawan: So for those of you who may not know about the meaning of life and all that, I think Dave is referring to this “Hitchhiker’s Guide to the Galaxy”. This term comes from there.

[00:16:20] Developer Productivity Challenges

Henry Suryawirawan: So you’re talking about increasing productivity. Is it still the case that developer is still not productive? I mean, technology has advanced rapidly, so many tools, so many advancements, AI/ML, and all that. So maybe if you can elaborate a little bit more, why do you think developer’s productivity is still a challenge these days?

Dave Thomas: I think because there’s just too much stuff. Vendors, it’s the API field of dreams. Every, you name it, the next Amazon or Google, cloud, they’ll introduce another 40 or 50 APIs. They’ll sort them alphabetically so they’re easy to use. There’re many cases, they do the same thing. So you can’t even tell which one should I choose. The problem is just there’s so much of this stuff. There’re so many libraries. There’re so many open source tools. There’re so many commercial tools. It’s very hard and they’re constantly changing. So if you look, people are, oh no, I’m moving from this one to that one. Or I’m upgrading from this version to another. So people are not using them. They’re learning the next one, and so it’s very complicated.

API design and framework design is really language design. And language design is very hard. When you’re in a rush and you’re producing APIs very quickly to be competitive, you often don’t get the same thought, documentation and examples. So this means that poor developers have to perform experiments. Well, the good news is there’s lots of things to blog because now you have to find how to use React 14, and the latest version of Go and Rust. These languages are actually very nice. Kotlin is a much nicer, simpler Java. But the surface area with these APIs keeps growing, and it varies from vendor to vendor. So this makes things very complicated.

The other thing is that, my colleagues tell me that we’re in an asynchronous world, and then we have to have everything be fully asynchronous to scale and get performance. This is definitely true. The bad news is asynchronous is not something that most programmers are really good at. There is a mechanism due to Eric Meyer, called Async/Await, which is heavily used. But there’s a bunch of very subtle state machines underneath that if you really want to understand what’s going on, you need to understand what these state machines are doing.

Asynchronous computing is still a challenge versus if you do something like Fred George talks about very simple data flow. Before you knew about it, there was a thing called data flow diagrams that were very easy to understand. Today, they are very easy to implement on a microservice architecture. But when you start building these asynchronous systems, they’re much more complicated. So complicated tools, complicated designs and very demanding requirements in terms of performance and scale. That’s why programming is harder today.

Henry Suryawirawan: I can see some of these challenges as well, especially for me, learning new tools, learning new technologies. You just finished one or two years working on this technology and then suddenly, oh, there’s a new one. Okay. Maybe we should give it a try. And the other advantage is always, there’s a new job for these people. So you learn something new and then you can jump to another job.

Dave Thomas: Yeah, absolutely. I mean, it’s great for resumes, right? You just put that new thing on the resume, if you can run ahead. And it’s a serious investment. So if you really learn it, I mean, everybody else has to learn all those APIs. That’s your 10,000 hours that you have to put in.

[00:19:37] Maximum Value, Maximum Speed

Henry Suryawirawan: So you mentioned as a subtitle of this project, 42D, you mentioned about a maximum value and maximum speed. Maybe if you can explain, what do you mean by maximum value? Is it that the software that we built has a maximum value out of it? And maximum speed, do you mean by the performance of the software? Or is it the speed of churning out that software?

Dave Thomas: The speed has to do with the speed of development. If it’s high value software and it has to be fast, then it needs to be fast too. It’s really the value of the software in the sense that we want to write as little software as possible, and we want it to have as much value as possible. I think that’s an obvious thing, but if you actually focus on that, then you think a lot more about how can I build that? How can I build that quickly? How can I have it maintainable and evolve? And how can I try and focus on the things that are maximum value versus often a project gets into all sorts of features and other stuff which really doesn’t matter. So it means that you have to be close to your customer.

I think this is obvious. But if you talk to most Agile developers and you ask them what the business value of the story is, they say, I don’t know, it’s just more important than this other one. And they’ve buried in little tiny stories because they don’t really have the big picture. That was really when Kent talked about the metaphor. He was really talking about having the big picture or the really big story in some sense. Because if you understand the key story, then that’s where the key value is, and that’s what you want to try and focus on and implement. Not get distracted into all the details.

[00:21:07] Big Is Not Better

Henry Suryawirawan: And you start with this problem, this challenge with many of the software development teams is that you mentioned the speed actually comes if you form a team that is small enough. You also mentioned write as little code as possible. Bring as much value as you can. But the tendency these days is actually to build more and more. Microservices, build more teams, add more people. There are so many developers these days, the demands, right? So maybe can tell us a little bit more. Sometimes this is obvious, big is not better. But the tendency for us in this software world these days is just to cramp and add more and more.

Dave Thomas: Oh, I think you can go back to Mythical Man Month where Fred Brooks, “No Silver Bullet”. This is the story of the building the IBM 360 computer. It is still a great book. He points out that over and over again, we have this nice triangle about resources, (quality) and time. Despite the fact that everyone who’s ever worked in software says, yes, we need a smaller team. Because we can move quicker. We can focus. We can maintain quality. Management has this panic because of everything rushing along today. And the fact that Uber has 4,000 developers and maybe he only has 3,500, so they need 5,000, so they can have more than DoorDash or whatever. So there’s just this desire to throw features out.

And this is particularly true because of the web culture. So web facing applications, people love to pile features in them. The problem is that once you get this many developers, you’ve just got more and more people basically sticking code onto code mountain, and it just becomes a bigger and bigger mess. Even though you’ve got tooling for testing and things like that, in the end, the only thing that matters is the code. And when code matters, then you get complexity. That’s when you get technical debt. The Agile developers, in the beginning, everything is great, and then they start begging for technical debt cards. Who created the technical debt? Them. Because too many people are running too fast.

The other thing is if you have to use a lot of different languages, a lot of different tools and you have to communicate between more teams, then you’ve got all this overhead. So you’ve got the latent complexity plus the accidental complexity, like the APIs are not working yet because it’s a brand new API from company XYZ. You just have more and more people and just piles up. I’ve not met anyone who has any experience who thinks that hiring a lot more people is the answer for how to get better software. But people do it.

Henry Suryawirawan: The tendency from the industry is like that. I’m working in a startup. So it’s also the same thing. Hiring is always a challenge. We want to hire more and more. We want to hire faster, and we have a lot of competitors as well. People also poaching these kinds of people. So it seems like what you’re saying is resonating with me as well. Many developers now are being demanded, and there are more and more features and code are being written.

Which brings us to the discussion that you mentioned. More lines of code is actually problematic. We have so many developers, maybe they build more and more lines of code. Why do you think bigger lines of code is always a challenge? Of course, maintainability is one. But what is the tipping point that we should say, okay, these lines of code is actually more than enough?

Dave Thomas: I think the first question is why are we building this? You’d be surprised how many things where there’s people working on this and having a big debate about the architecture or the technology or the features. If you stand back and say, why is this important? What customer cares about this? Often, there is no customer that cares about it. It’s just somebody in the organization thought this would be an interesting idea. The easiest way to reduce the code is not to write it. Is what we’re getting really essential? Is there a way that we can simplify that into a mechanism which is much more compact? So essentially we can replace something that’s hard-coded with an interpreter, which is easier to change. Clearly, the number of developers influences this.

From my point of view, I think that if you have an IDE, it shouldn’t allow you to scroll. That will limit the size of a function or a method, so it has to fit on the screen. Maybe you can get smaller font if you want larger methods. As soon as you get into a world where you have to scroll, or basically you have to use Emacs to find where the functions are, and so on, I think you just get lost in the complexity. Even the tooling, right? I mean, you get the latest release of Visual Studio or Eclipse. It really doesn’t matter. There’re more knobs and dials and ways to do things. So you just put more and more complexity onto people.

And often, a lot of the functionality there you don’t need. It’s not obviously layered. It’s there stuck off in some corner some place. There’s a whole library. You want to use a framework, but it turns out you only need two classes in the framework. But as soon as you pull the framework, it injects all these dependencies into your program, because you have to know what’s inside the framework. So it’s not really a component. It now becomes part of your program. Frameworks were a big mistake of OO. The idea was you were to use a framework inside and then you were to have an external API, which was just a value-based. So you didn’t actually have to know what the classes were inside the framework and so on. So you didn’t have to subclass them and so on. But when you use a framework, you get too intimate with the actual framework, and now you just create another big dependency. And so many people’s dependencies actually aren’t their own. They’re dependent on, oh, look at this great library that I saw. Look at NPMs for example.

[00:26:26] Tools Getting More Complex

Henry Suryawirawan: I mean, like you’re talking about the modern maybe technology disease as well. So many libraries. So many open source tools. So many things that could be easily installed these days. It could be like NPM install or whatever. You’re categorizing this as modern tools becoming more and more complex. It seems that the answer to a certain problem is to just build more stuffs. I’m quite fascinated by this idea because yeah, there are so many things being built these days. Even like, if you look at JavaScript library, maybe every few days there will be one being created out of any corner in the world.

Dave Thomas: Good developers try to use as few libraries as possible. That’s basically the secret. Do we really have to use this framework and this library, because the experience is that those things have a high cost down the road. It looks like an easy path. But often they may go dead. The project may fork. So definitely, too much code is definitely the problem, and in part, contributed to that with the whole notion of frameworks and inheritance and so on.

And stateful programming. One of the benefits of functional programming is that you’re much more limited in the sorts of things that you can do. And in general, that’s what we really need. We need fewer ways of doing things. For instance, when Amazon came out with their first APIs, they were very simple. Basically, you just had S3 buckets. API was very limited. But the good news is it was easy to use because there weren’t many ways to do things. Now where everything you do on AWS or any other cloud platform, there is at least five other ways to do it.

Henry Suryawirawan: But, I mean, the counterargument would be people create these tools with the intention of increasing productivity, right? More frameworks, more abstractions, more libraries is actually encapsulating all those unnecessary details that probably developers don’t need to think about. So what about that angle? How do you counter this argument? Oh yeah. We’re building more and more this, for example, AWS services so that developers become more productive and they just do things with a simple API call, and then we take care of the rest.

Dave Thomas: I think it’s a nice story. But most of them are written way too quickly. They may be well-designed, but they’re certainly not well-named or documented. So I don’t buy that story. I think the answer is, if something is really good, it’ll make its way into a library that’s used and not into a big tool chain. You just look and see, look. Okay. Now, because we’ve got event-oriented programming, we don’t have a stack anymore. So what you have to do is you have to turn on tracing and then you guys have to get yourself a Honeycomb or some other observability (tool) so you can try and create a stack from this event (trace). Something’s wrong here, right? It’s just too complicated. Maybe I’m just old and don’t get it, but it just seems like it’s just too much stuff.

Henry Suryawirawan: So I had an episode the other day with Mark Seemann. He was referring to cognitive load. Programmer’s brain these days is really super, super intense because there are just too many things. So it could be the states, reading the code. It could be the tools, understanding the APIs, the surface areas that you’re talking about. It could be also all this orchestration, observability tools. It seems like these days, if you want to be trendy developers, you really need to master a few of these technologies before you can be considered the top level of developers in the software world.

Dave Thomas: Yes. Definitely the case. On the other hand, I think there’s lots of smart companies that are looking at things like low code. Low code is not something that all the aspiring hot developers want. And that’s good. Because you want them to go some other place where they can have that big mess at that place with all their tools. So I think for normal businesses, low code tools are a good approach. They reduce the surface area of what you have to use. They constrain what you can do, which is a problem if you want to produce the most amazing JavaScript website or microservice architecture. But it allows people to get things done fairly simply. So I think the economics for companies that are willing to go with a low code approach are very promising. There’s a lot of choices today.

Decades ago, that were things called 4GLs, which were the low code languages at the time, and people were very productive in them. Now people look down at them because real programmers don’t program in low code. In the end, if you’re running a business and people are successfully knocking on applications, that seems to me like a pretty good idea compared to trying to figure out how you will hire the bright guys who argue with each other over which technology and microservice architecture they’re going to use and so on. I think simpler is always better.

[00:30:45] Importance of Domain Knowledge

Henry Suryawirawan: So one thing that probably does not get simpler is the concept of business logic, right? Depending on the industry, of course, these days we need to build more and more highly sophisticated rules, logic and whatever that is. You mentioned that the domain knowledge is actually still important. So this also goes back to the concept of Domain-Driven Design and all that. So tell us more why you still think domain is still the key here? I think the challenge here is how can we get more developers to care more about domains?

Dave Thomas: When you used to start in a company a long time ago, you would actually had to work in different departments. So you would actually find out what purchasing was like and what inventory was like. And you’d meet some people there. Those people were your links to the domain. But now developers, they like to only focus on coding. They don’t want to know any of that domain stuff. “Look, just give me the story.” But what this does, it burns out the product owners or the product managers, because they have to take something that’s a very simple domain concept and break it down into almost a task level so the developer understands it. And to me, if you’re breaking things down to task level, you’ve missed the point of Agile. Basically it means you don’t really have communications anymore. You basically are forcing people to break down all the work into tiny things that developers can implement.

I’ve worked for First Derivatives, and one of the interesting things they did is they hired graduates out of science and business, and they would teach them capital markets. Essentially everything about stocks and options, the domain, and they would teach them how to program in the q, a (vector) functional programming language. But to the employers, they were super attractive because they could go in to a company like the stock exchange in Singapore, for example, and they could be productive because the business people could communicate with them. They actually knew the difference between a swap and an option, what the rules were and so on and things like that. So it really allows efficiency in the organization. Because now the business and the developers are speaking the same language. So I think it’s a very, very important thing to train developers in the domain because you cut down all this overhead.

The other thing is that there are techniques like decision tables, state tables, spreadsheets that work really well for business people, and those can be automated into code. So instead of writing a bunch of story cards and then coding them into Java or Go, you can automate these. They’re just data-driven so you can change them. These are very old techniques, but they work very well, and they’re very good for capturing business knowledge. Most people in the business can put their rules in a table or they can put their calculations in a spreadsheet or they can put their workflow in some sort of state machine or state transition diagram. These are generic techniques that are very powerful. Yet many programmers aren’t taught them in school. They’re not familiar with them, and so they end up hard coding all these things. So the calculation changes, you should just be able to change your spreadsheet and deploy and run it. Today, we can run spreadsheets at blazing speed over trillions of rows of data. We don’t need to hard code that in Java per se.

The other thing, if you talk to Eric Evans, I don’t think there’s enough emphasis on domain language. If you actually capture domain language, in the old days we called it a data dictionary. If you have essentially a well-known vocabulary with definitions and so on, those names travel through the architecture and the code so that you can actually relate the words in the requirements and the words in the stories, and in the code and the architecture. You can get the overall flow, so you can find out what happens versus if you just go into a code base, you find programmer names. They’re either XYZ or some horrible elongated thing that you can hardly type, and that takes up all the space on your screen. So what you want is to reflect the domain in the code so that it’s easier to understand. So companies that want to make it go fast, they invest in making sure that developers understand the domain.

If you’d be ever going to be in a business, transitioning from a developer to someone in a startup, you learn very quickly that speaking the language of the customer, speaking the language of the domain is absolutely critical to success. The customers don’t come to you, then describe exactly what they want. They expect you, as a founder, to have this vision and to be able to articulate it to them in their own vocabulary.

Henry Suryawirawan: I have two things that I want to ask you as well. Firstly, it’s about working in startups. A lot of startups, actually, although some work in well-known domains like financial technology, medical, but maybe the founding members are not super business experts. So this knowledge actually is not there. What will be your suggestion for these people? They have the passion to work in a startup, building products, but they don’t necessarily have the business experts. So they hire product managers, which are generic product managers, and having to build a specialized domain software. So this is, I think one of the key trends these days, why probably software product is not optimized.

Dave Thomas: I definitely agree with you. I think, the problem is generic programming, a generic product person, generics are really never all that useful. In the sense that what you really want is someone that has some value. They have experience in building real time control systems, and that’s why you’re hiring them. You’re hiring them for their expertise. The way in which we like to do it, where you can, is work with domain experts.

So if you’re going to be working in a particular application area, what you want to do is find those domain experts. They’re more important because what you need to do is get steeped in the domain. So it’s better to engage with two or three real domain experts. There’s someone that’s retired, maybe, or they’ve done very well, and they’re able to spare you their time. But you can essentially learn from them and build your kind of domain model and your understanding before you get your startup going very far.

The problem is once you start coding, you’re in trouble. Basically, then you’re already committed. You’ve got a schedule. You’ve got a release plan. So, it’s all about investing in the right places before you start coding. Because the more you can define, describe in words and in the architecture, the more you’ll be able to add new people cause they’ll come in and say, oh, okay, I see this. Just play the movie or the talk about how we got here and where we’re going and what’s the domain about.

Companies that were very good at this, for instance, ThoughtWorks started originally, a very successful consulting firm, I’m sure you know them. When they started, they had this unique idea of having really super business analysts. People who really understood given domains. And that was one of the things that was a real major contributor and impressed me very much. When I met Roy and his team many years ago, they had business analysts who had decades of experience in the given industry. So when they went to work with a customer, the ThoughtWorks team had the benefit of this person that really understood the domain, had the whole vocabulary, could relate to the customer. And I think that’s something that you really want, someone that has experience and understands the customer. So if you can bring someone like that into your startup, whether it’s someone that comes in as a resource in the evenings or the weekends, and say, “Look, what is this? General ledger thing, right? I hear they have this in accounting. What is it?” It really helps you. That context is very important. And people don’t invest enough in understanding domains.

Henry Suryawirawan: Yeah, I think I agree with you. I mean, many people, many software development teams do not invest enough in understanding business domain. Because you mentioned about as soon as the code is written, right? Not only that. As soon as the investment comes in, as soon as the turnover is starting or as more and more people joining the company, then it’s a total mess because there’s no clear picture of what the software is doing. There’s no same understanding about the domain. So I think that is the current challenge of software industry.

Dave Thomas: Yeah, I don’t think you should take much money until you’ve actually figured out what you’re doing. That’s not the tendency, but most of the businesses I know that have been successful have actually spent a good deal of time early on thinking about what they’re doing, understanding it and not taking money. Because as soon as you get VC money then you’re on the wheel. You just have to run faster and get more done, get more customers and it just doesn’t end.

[00:38:53] Decision Tables and Spreadsheets

Henry Suryawirawan: The other thing that you mentioned that I was fascinated about, because last time I used to work in insurance industry, I was working with actuary people, building this point of sale system where you calculate the benefits and all that. And they all use spreadsheets. These decision tables, calculators, and all that. And we, programmers, have to convert that complex spreadsheet into code. Which sometimes when you look at it, you can’t even tell, okay, this calculation is actually as simple as a spreadsheet calculation. So I think what you mentioned here, maybe we could just leverage our decision tables or spreadsheet even, to build programs instead of having to write lines of code in general programming language. So maybe can explore a little bit more here because I rarely see people doing this.

Dave Thomas: Take a look at things like Airtable and so on. There’s a whole lot of these sorts of low code systems, and you’ll see that they’re really about spreadsheets. I mean, in the IBM analytics environment, we did the analytics in a spreadsheet. So even though you’re computing over a trillion rows, it was still managed in this spreadsheet. You can actually just run spreadsheets. There’s no reason to give them to a programmer to get them wrong. It’s a very straightforward algorithm. You just have to do a topological sort that puts all the things in the order. And then put the code and you can write a simple interpreter. You can write one Java program and it’ll run all the spreadsheets.

In fact, years ago, Kent Beck worked on a project. It was a Smalltalk project at the time, but a big insurance company in Chicago. They were trying to replace COBOL programmers with Smalltalk programmers. It went okay, but it didn’t go nearly as well as the Smalltalk people argued it would. This was the early days of XP. But then Kent observed that the people who negotiated the contracts and the insurance policies actually did it all in spreadsheets. So they actually implemented a spreadsheet interpreter and put it inside the mainframe, talking to the COBOL programs. They did it in Smalltalk and they also wrote a rule checker because spreadsheets are usually a mess. So before they deployed the spreadsheet, they first of all had a rule checker so that the analyst would be able to do this. So here’s a case that two species of special purpose tooling, and they were able to automate.

If you’re going to do concurrent state machines, so you can do this high-speed messaging, asynchronous microservices, you have to build state tables. You have to be able to understand those. Clearly, you can do the simple ones with step functions in AWS. But if you’re building a complicated system, you’ll have to build your own state machines and your own interpreters for those state machines. So, this stuff still works. It’s been around for decades.

Henry Suryawirawan: Yeah. Spreadsheet, I think, is still the most important tools this day. Because the interface is simple enough. Many people understand. If you translate to code, you have to do testing. You have to build UI. You have to build so many things so it becomes a challenge.

[00:41:39] Importance of Programming Languages

Henry Suryawirawan: So another thing which you mentioned. I know that you have been dealing with so many different programming language, including building some of those. You mentioned that programming languages actually matter. These days, there are plenty of programming languages. They are probably too many to learn. So tell us more, why do you think programming languages matter? And how we should choose it, actually?

Dave Thomas: Choosing is much harder than the other question. In the end, certainly there are people who disagree about this. Some people think that you just do the design and the programming language doesn’t matter. I know Ivar Jacobson, his view is that you just get the system design right. But in my experience, the more code you have, the more problems you have, the more developers you need. So an expressive programming language really reduces the amount of code you have and increases the rate at which you can make change. If you have a small code base, it’s very easy to make changes so you can evolve the code quickly.

If the language is simple, then people don’t have to debate which library to use, which API, if there’s only one way to initialize something, it’s simple. If there’re five ways to initialize it, you have to go, which one should I use? And why should I use this? Most of the, I will call them exotic programming languages, things like Haskell or J or K or Erlang, for example, have very expressive ways for handling the kinds of problems that they work on. No language is ideal for everything. So a language that’s simple with regard to syntax and semantics, is going to be naturally much more productive. The other thing is it’s likely to have a cleaner implementation. Because the number of cases that the implementers have to work with is much smaller.

The advantage of languages, which don’t have the library written in the language, is you don’t get this interaction between the runtime and the library. Java has all its libraries written in Java. The bad news is that this means that a change in the JVM may not be reflected in the actual libraries. When Lambdas were added. You have to change one thing at a time, so I’m not criticizing the Java implementers here. To be able to do handle things like MapReduce efficiency, you can put the mechanism in the JVM, but the library doesn’t use it. So you have this codependency between the library and the runtime, and you have to get both fixed to do it. Versus languages where the library essentially written in C or machine code. The library can be very tuned. Versus when you look at Python. Python is not my favorite language because I don’t like whitespace. But the language actually looks like it’s fairly fast. That’s because it’s actually very good at using the same old libraries. Some of which are even implemented in Fortran. Pandas rely on proven libraries that are not given to you in the source code.

So I think the advantage of those languages which separate those concerns. I don’t think it’s always a great idea to have the library for your language written in the language. Because, again, language design and library design is hard. If you make it easy, people go, oh look, I can change what object means or some nonsense like that, or change hash or something like that. You don’t want people making those sorts of changes. I think it’d be good if everyone got the source code to read for the libraries, but they weren’t allowed to change it. Source code is very important. It’s just, you don’t want everybody changing all the source code. I think choosing a language which is simple and expressive. That is you can learn it quickly. It’s got a simple consistent set of rules, has a good implementation. You can run a benchmark to see, is it fast enough?

So if you’re choosing a more exotic language, you need to address those concerns responsibly so that you really understand that this is going to work as what you expect. And it probably means that if you’re using exotic language, for instance Erlang, lots of code in the Ericsson switches which is written in C. But that’s perfectly good. They’re small functions, which do what C does. The overall switch architecture, which is sort of the penultimate microservice architecture, is well-designed and Erlang supports that. But for some of the critical code sections, they do the right thing. They use another language. They program the critical code performance wise in C. So I think that’s a good separation of concerns. There’s no point using Haskell to try and compete against C or Rust. People will do that. But if you want to express something that’s got some very serious system of types and algorithms and so on, Haskell is a beautiful language for doing that. You can always use some other mechanism for the low level pieces of code.

Henry Suryawirawan: So, out of curiosity for Dave Thomas, right? If you have to choose one popular or enterprise language and one exotic language, maybe if you can share with us what will be your two choices?

Dave Thomas: Two choices. I think probably for an enterprise language today, I would choose Kotlin. It’s a simpler Java. It’s close enough to Swift, so you can get by for doing mobile for at least Android and so on. So I would say, I think Kotlin’s a nice piece of work.

Henry Suryawirawan: And one for exotic.

Dave Thomas: K.

Henry Suryawirawan: I don’t even know what that is, but okay. I’ll check it out later.

Dave Thomas: I’ll send you the link.

[00:46:46] Data-Centric Approach With Legacy

Henry Suryawirawan: Sure. Another thing that you mentioned about working with legacy, right? So a lot of systems this day have been built for many years. They are legacies. They maybe have been written in COBOL, Java, whatever that is. These days we still have to inter-operate with them. You mentioned something that is, to me, is quite insightful. Instead of doing all this rewriting, why don’t we just integrate by using a more data-centric approach? So, if you can tell us a little bit more, what do you mean by this?

Dave Thomas: Sure. I think everyone went to the Michael Feathers' school of legacy transformation. He and Michael Nygard and many of the great YOW! speakers talk about strangling your monolith. But strangling is very expensive and takes some very talented people. It works. But it can take two or three years to take your monolith and strangle the pieces and turn it into services. The approach that I’ve advocated and many others as well, is that changing code is hard. A lot of the times you don’t even want to change it because it’s working fine.

So the real issue is you want to introduce new performance or functionality, new capability into the system. There is a natural place to do that. That’s wherever the data is stored at rest or moving along. So basically, this can be streaming data. It can be a persisted file or a database. It can be a place where communication happens between programs. This is an easy place to have an intervention. It’s an easy place to insert a new technology. The nice thing is you can essentially put it in there, right into the existing systems and divert the flow. So you can just take some subset of the transaction so you can test it. You could take like 5%, verify that they’re working and so on. You can select the high-volume transactions and say, okay, we’re going to process these differently because we have too many now. Our system is too slow. So it’s very easy. And the nice thing is it’s easy place to insert new technology.

So you can use, for instance, one of my favorites is in-memory database. So essentially, you can do things blazingly fast in terms of queries and so on. That’s essentially what Honeycomb provides for doing observability. They have an amazing database that you could do queries over to essentially try and recover the stack and figure out what’s going on. The nice thing is they’re fairly easy to do, and you can usually do them in three or four months. So you can complete the whole project by putting this new technology in place. You can put new hardware in place. Intel just announced a new chip. But basically, you know, for $2,000 you get a terabyte of RAM. Another $2,000, you’d get 20 terabytes of NVME memory. It’s got 64 cores, I think for $8,000. So, you can get massive compute capability and you can take that and you can insert it in the appropriate place. You can use it for testing. But these things are all fairly straightforward. They can be done by a small team. By carefully looking at the data flows, the data provides a natural place where you can check and verify. So testing is much easier as well.

You can just think about decoupling the data stream and putting this inside and then using it. It’s a very powerful technique. I’ve used it for all sorts of things. For instance, high-speed database refactoring. Taking a database from one technology to another technology just by streaming it full speed from one representation to another. Very different approach to how one typically does this. Just trying to get the data out using SQL is a nightmare. So this thing actually takes the pages right from the physical store and transports them to pages on another physical store in a different representation. But the actual algorithm is very straightforward. Its simple algorithms and the access to modern hardware really lets you make huge changes in an organizational system without disturbing very much. Since it doesn’t disturb very much, there isn’t a big issue about DevOps changing the data center, doing all that sort of stuff. You were just saying, look, we’re going to take this and we’re going to put it in a different format.

[00:50:35] Future Programming Is Query

Henry Suryawirawan: Another thing which I learned. Many people think, especially in the microservice world, data is actually an internal to the service that should not be shared or, you know, you should not even care about what columns, what data types and all that. What’s your perspective about this? If you mentioned about integrating using data-centric approach, we need to understand those internals of the data structure. So what will be your advice here?

Dave Thomas: I’m definitely wrong on this. So I’m happy to be wrong. I actually think that the future of programming is query. The whole issue of having a 140 REST interfaces, when in the end, most of those things are just queries. So if you look at GraphQL, I would actually go further and said, I think there should be a web SQL where you can just write a query and you should never have to see a REST and the things like that. REST can be underneath. I understand the properties and so on, but I believe the data centricity is still a very valuable approach. I understand why people do it, but it’s like the same thing. People don’t even want to reuse any code. We don’t reuse any code. We just write whatever we want. So now you’ve got 13 or 15 versions of the same function. All different because we’re going to move fast. Well, in the end, you’re going to have a lot more code and a lot more code interfere. So I think you have to use your head. It depends on how often people do this for performance or isolation. But in the end, someone still has to understand where all the data is. Now the theory, of course, is it’s all moving, so you don’t need to worry about it. I’m not yet convinced that when you look at the architecture diagram and you see that everything talks to everything. Does that really help you understand if you see those wonderful microservice communication patterns? Is that a good thing? I don’t think so. Everything talks to everything.

Henry Suryawirawan: Even sometimes you don’t have the diagram. You see chatter of logs over multiple services.

Dave Thomas: Well, that’s why you have to have those tools. So you can get the diagram. Look, it’s always good to be able to isolate things and separate them. The problem is sometimes that can actually contribute more to the complexity. It’s sort of the same thing as NoSQL. NoSQL looks like a free lunch. I’m a Ruby programmer. So I just stick it in a blob and I don’t worry about it. It makes it easy for you, but every person that wants to look at that data has to write a query to unpack that blob. So what you did is you traded your convenience as the developer, by stucking in a NoSQL blob with the pain of everyone who has to actually use that data, having to parse it and take it apart. I think we need to pay a lot more attention to how to write queries. In fact, if you look at all the new tools, observability of Honeycomb and some, they’re all taking us back to. Look, if you want to understand what’s going on, you’re going to have to query all this data, all these traces.

I think query is kind of the underrated tool. And I think it’s partly because programmers hate SQL and don’t like databases. But the nice thing is there are databases which are functional technology, which are very powerful. And I think as you see and use those things, you go, “Oh wow. I didn’t actually have to do all this other stuff. All I had to do was write a query.” You can even do points to analysis and compiler concerns. Like if you look at the Datalog or was this a talk at Lambda Jam by Simon Marlowe from Meta (Facebook). They’ve just built a system for analyzing programs and essentially it’s all done in Haskell, of course. They basically have built the data log engine so they can query properties about programs to try and understand what’s happening in different programming languages and interaction inside of Facebook. So, I think query is the dark horse for the future.

Henry Suryawirawan: Everyone is moving back to SQL as well. So really looking forward to see the day where we don’t need to learn so many different query languages.

[00:54:14] 3 Tech Lead Wisdom

Henry Suryawirawan: So Dave, it’s been a pleasant talk. I always learn many new things. So it’s insightful. Not only that. Some are unintuitive for me. But those things actually make sense after you explain about it. Unfortunately, due to time, we need to wrap up pretty soon. But I would like to ask you one last question that I normally ask for all my guests, which is about three technical leadership wisdom. So this is like an advice for you to all technical leaders here, what will be some of the lessons that you want to share with us?

Dave Thomas: Scary. Three questions. The first one is actually a quote from a very famous gentleman called Seymour Cray, who invented Cray computing. “Hire them young. They don’t know it can’t be done”. If I’ve had any good fortune, it’s been to hire many young people. Often just in their second year of university where they’re very innocent, but enthusiastic, and they do amazing things. If you hire them with a PhD, they can’t do anything because they already know all the wrong ways. The great thing about young people is they are not afraid. So with discipline and guidance, obviously. They do amazing things. Don’t underestimate the young people that you have access to. Singapore has an amazing pool, so exciting place to hire.

The next one is focus isn’t just important, it’s everything. The most important thing you can decide every day is what you’re not doing. Everybody has a big list. So the real question is what is that you don’t need to do today? What are you not going to do? Those are the big decisions. Very simple, but hard.

The last one is that every release should have less code in it than the previous release. If you’re a really great software organization, you will figure out how to get more function in the release and reduce the code. May only be 3% or 5%. But if you can add new function and delete code, you’re probably on the way to having a maintainable code base. If your code base keeps growing, then you have a big maintenance problem and lots of challenges ahead of you.

Henry Suryawirawan: I really think the last one may be difficult because as we build more features, normally more lines of code is being written. But thank you so much. And I love the focus question, right? Because everyday we should think about what we should not do today. So I think it’s always a good question to ponder.

So Dave, if people want to follow you or continue this conversation with you, is there a place where they can reach out?

Dave Thomas: Sure. They can go to my Twitter, @daveathomas or they can just email me at dave@bedarra.com. Thanks very much, Henry. It’s been really great to talk to you as always.

Henry Suryawirawan: Thank you so much for your time. Really pleasant conversation.

Dave Thomas: Take care.

– End –