#269 - The IT/OT Handbook: From Silos to Sustainable Digitalization - David Ariens & Willem van Lammeren

[00:01:57] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today we have two guests, co-authors of this book from IT Revolution titled The IT/OT Handbook. So if you don’t know, IT/OT is actually standing f- stands for, information technology and operation technology. I’ll let them later on explain, why there’s this IT and OT. It was a bit confusing to me what is OT in the very beginning. But yeah, looking forward to learning from both of you because I think these topics might be interesting, especially, for us to, you know, understand what’s the challenges when working with operations. So welcome to the show, David, Willem. Looking forward for, you know, this conversation.

David Arens: Thanks for having me.

Willem van Lammeren: Same.

[00:02:42] What Is the Story Behind the Book and the Podcast?

Henry Suryawirawan: Yeah. So I know both of you also have podcast, right? So maybe this is also an interesting time for you to be guests of a podcast. But let me start by inviting both of you to share a little bit more about, you know, the story behind the book, right? And maybe also the story how you came up with the podcast, because I think it’s kind of related in a way.

David Arens: So Willem, we need to be careful to not start asking questions to Henry this time.

Willem van Lammeren: It is a bit different to be on the receiving side.

David Arens: Let me start. So a quick introduction. So indeed, yes, I’m David. My background is mathematical engineering. I started working in the chemical industry, 15 years ago. So I always worked, let’s say, between all kinds of fascinating chemicals, but always on the digital side. So never really touching the actual production of chemicals, but always trying to, yeah, find ways on how IT can help with the production of those chemicals. And that’s, yeah, where me and Willem met 10 or so years ago, maybe more.

Willem van Lammeren: 15 years ago, I think. Something like that.

David Arens: Yeah, absolutely. And so we, but I leave the introduction to Willem as well. But, this is where I think we started working together on this industrial digitalization topics. And this is where I think the, we can already say that OT, it stands for, yes, as you said, operational technology, I think Willem, and that’s maybe the perfect way for your introduction as well. You always said, “David, it’s just IT for factories.”

Willem van Lammeren: Yeah. Yeah, that’s the problem is, nobody knows what I do except David and a couple of other people. So I’m Willem, the other half of the IT/OT Insider. I’m a bit older than David, so I’ve been doing for 20 years working in manufacturing all my career. Stumbled into IT. First, you know, the really corporate IT far away from production. But when I started working with David, it was like on a huge site, plants, production sites, pipelines, everything around you. There was no denying we were in production doing IT for factories in essence, yeah. My background is a chemical engineer. But I mean, it’s been so long ago that I can only pretend that I’m an engineer by now.

So, you asked us where did the podcast come from? Where did the book come from? I think it started out me and David, of course, working closely together on really cool projects in that intersection where, you know, the digital meets reality. Because you can talk about programming and codes, but there’s also stuff like pumps the size of an apartment building. And that’s where we’re working. And as we both had different paths in our careers, we still kept in touch because we’re frustrated. And I think frustration is the reason why we started writing. It’s like saying like, “Why don’t people do this this way? Why don’t they understand it’s working differently?” And that’s why we started writing, because I think writing creates more understanding.

David Arens: Also there’s a very good reason to just keep on meeting at Willem’s.

Willem van Lammeren: Also, also, also. Plus nobody else would understand my frustrations anyway.

David Arens: We had to share them to each other, Henry.

Henry Suryawirawan: Yeah, so I think, David also shared with me, before this recording, right? So you guys started to write simply because of sharing this frustration. So it was like a therapy, you know, for both of you to actually kinda like share you frustrations…

Willem van Lammeren: That’s how it started, yeah.

[00:06:31] What Is the Main Difference Between IT and OT?

Henry Suryawirawan: … share it with the world. But somow it got picked up, right, by many readers. And I think it became something like good enough to write as a book. So why do you think maybe in the first place? Because many of my listeners are more from the software engineering IT side, right, what you call IT side. So tell us what’s the main difference of IT and OT? Actually, I’m quite interested by your definition by saying OT is just IT for factory. So tell us a little bit more because we are probably not familiar with this.

David Arens: I think it starts, and Willem already touched upon that, it starts with the fact that in our world, in the OT world, we are creating software which has a direct physical impact. So there is physical asset. And let’s just give it first, a couple of examples, like the types of industries we’re working in. I think that makes sense also for your audience. We often say that we work for manufacturing process, process plants, and infrastructure or critical infrastructure. So for example, critical infrastructure: water, electricity, that type of plants. It’s very interesting, for example, a water company, they have an obligation to provide clean and safe water at any time to their customers. You can’t reboot the water system. The water needs to keeps on flowing, always. The same for manufacturing. It can be cookies, it can be cars, it can be dishwashers. So all these things, they are getting manufactured. And yes, there are people there, but the digital side, the way these processes get optimized, automated, it’s becoming more and more important. And then we, yeah, we also touch upon the process plants, so we both come from a process background, so chemical plants, oil and gas. So these really, really huge, gigantic, yeah, city-size locations where you sometimes only have a handful of operators, operating a plant, which is, yeah, which is the size of a town, right? And this again, is where this digital component comes into play.

So I think this is the most important thing is that we’re dealing with a physical asset where safety is the most important thing. We always wanna, we always wanna make sure that these plants, they run safely. And after safety, comes the integrity, making sure that what we do is right is right, is for, it’s safe, it doesn’t explode, or the products are safe to consume or et cetera. And only then comes the, I would say, the availability of the plant and finally the confidentiality of the stuff we are doing.

Where in the IT world, it’s often a bit, I wouldn’t say totally opposite, but typically, we’re talking CIA – Confidentiality, Integrity, Availability. And we also have this possibility to do, yeah, easier integra- easier iterations, easier tests, try something out. Deploy a small, small change to a part of our users. That’s something, we can’t do in our world.

Willem van Lammeren: I think an easy way to get a gut feeling of this is that you wouldn’t want move fast and break things to be the motto in a nuclear plant. Or in a pharmaceutical company where they’re making drugs that you’re gonna take, and put in your body. So it makes sense. The problem is, of course, both are right. I mean, it would be terrible IT to work like you would in a nuclear plant. I think we tried that in the years, in the 2000s. It was super slow. It was frustrating, over budget, over time, nobody happy. And of course, operation is also right. You do need to operate the way that David mentions in production. The problem is we need to meet in the middle if we want to digitalize production. And there’s no solution. The solution is not gonna be telling one to be different. We’re not gonna tell IT to be like production, and we’re not gonna tell production, why don’t you be more like IT? And that’s, feel that tension, that’s where we operate in.

David Arens: And that’s a very good, that’s a very good point. This is exactly also where the book lives. The book doesn’t try to make the other change. It also doesn’t try to apply IT paradigms, I would say 100% to OT and OT paradigms 100% to IT. It’s really like, okay, how can we change that middle ground? And how can we do it in such a way that we can really, yeah, take these leapfrogs, these big steps which our industry really, really needs. So let’s be honest, I think we do need to take those steps.

Henry Suryawirawan: Yeah, so I think very interesting to understand these challenges, right? So you mentioned safety is, I think, the most important thing. So obviously I come more from the IT side, software engineering. You know, we build digital products. Not too many times we actually care a lot about safety, to be honest, right? It’s more about, you know, building, building features, let’s say.

Willem van Lammeren: Physical safety.

[00:11:58] Are Software Developer Skill Sets Interchangeable Between IT and Operations?

Henry Suryawirawan: Yeah, testing in production. Oh, there’s a bug. Okay, maybe we can fix that, right? So it’s very rare to actually put security, like, you know, like at the top, like really, really top of priority of what we build. So I think looking at these two different paradigm or maybe set of importance, right? Do you think like skill set might be different? Like, don’t know about the JD of soft- maybe software developer for operations, right? When you build all this machinery kind of a software, do you think like the skill set can be interchangeable or it’s something completely different?

David Arens: Willem, I think that’s one perfectly for you because you’re living more on the IT side now.

Willem van Lammeren: I’ve always been living more on the IT side than you, David. I think in essence, there is the engineering part where you do find each other, which is about trying to understand systems and processes and how do they work and interact. So I think that part, the technical part is very interchangeable. I think one thing that’s very different between a normal IT user relationship in a company, I’m not saying like a SaaS provider or something like that, but like internal IT in a company, is that OT is also a technical department. So unlike if you’re talking to HR, you’re gonna talk to engineers, and they love to geek out too, and they want to understand. They want to know how that AI classifier is working. And they would love to explain to you how pumps can fail in 17 different ways. I think if you bring that engineering curiosity to try to understand what’s happening there, you do have the basis for a good relationship. It’s important to acknowledge, though, they’re right in their technical expertise, but you’re also right in your technical expertise. I think very often IT is seen like unicorn dust and just program it. But if both sides have that mutual request, respect, I think you do have a lot of overlap.

David Arens: It’s an, yeah, it’s a very, it… I, I think, Henry, that the, well, so the understanding that you are working with two technical domains is the first one. I think a second one I’d like to add is that in a typical setup, in a typical company, let’s say in 95% of the cases, IT has a reporting line to the CIO or a similar function, and OT is typically part of plant operations. So they are part of the, yeah, they report to a COO or similar function. And that’s a, it makes sense. It makes sense that there is a centralized IT function and typically a localized operation function. But it makes collaboration also really, really hard if the CEO the company is the only common link.

Henry Suryawirawan: Yeah, this is also something that is uniquely different, right, in your world, right? Because the OT reports to COO. This is also something that I just learned when I spoke to you before this recording, right? So I always thought that technology always goes to CIO, CTO, and all that. But maybe in the industrial world, right, in the manufacturing world, actually COO is the one who actually governs all this technology, right?

[00:15:23] How Did The Phoenix Project and Team Topologies Inspire IT/OT Cooperation Patterns?

Henry Suryawirawan: Which brings to this kind of like silo I think, right, between these two team, especially if they have to collaborate with each other. Maybe, you know, you have this manufacturing process and all that, but also has the digital software aspect that interacts with each other. So tell us what’s the problem with this silo, right? And do you really still see so-called in like the DevOps world, you know, throw over the wall kind of mentality? Maybe a little bit on this.

David Arens: Yeah, yeah. It’s, you know, if– I wanna take a step back first because you already mentioned our book is getting published by IT Revolution. Like that’s already a dream came true for, for, for multiple reasons. A, because they just wanna work with these guys from operations that’s already like, wow. But secondly, if we go back many years in time, at a certain point, Willem advised me to read The Phoenix Project. And I think most people in IT, I wouldn’t say everybody, but most people in IT will have read…

Willem van Lammeren: I hope they did.

David Arens: Yeah, will have read…

Willem van Lammeren: Or should read.

David Arens: Or should read The Phoenix Project because it’s still like the, for me, it’s the starting point of DevOps. So I remember, I still remember me reading it. I actually read it already a couple of times just to get acquainted with the content as good as possible. But I know I read it, and I became nervous while reading it because I was going through at a… it’s also, you know, the background is also a manufacturing plant. It’s car, it’s a car parts plant. But I was reading it and I went like, “But yeah, but we, like, we face the same type of problems.” And then a little bit later, I also read, again, or I think Willem, you pointed me to it, it’s, that’s Team Topologies. And you also go like, “Yeah, but this, like, really makes sense.” But then, and this is the interesting thing, then we started talking about can we actually apply DevOps principles to those siloed worlds? And if so, how?

Willem van Lammeren: I think that’s where we started to experiment, David, because just for context, when we were working together, it wasn’t just me and you, David. We had like a bunch of people working in that domain with us. So our job was how do we make sure that these group of people who are working with servers every day, patching, doing code, working with database, I mean, they were doing IT work for those factories. And the ideas from DevOps about how development and operations in IT could work together, we tried to apply them between IT and production environments who were our internal users. And we found that given a few tweaks, actually those were very translatable in terms of concepts. So we managed to get those same ideas and concepts working. It changed also a bit how we see how you run teams. And again, adding those tweaks for operations specifically is a bit the basis of the book, I’d say.

David Arens: Yeah, yeah, yeah. Yeah. And then we stopped thinking about it for a couple of years. And then like, and I think that’s, that was perfectly fine because, you know, you’re just trying stuff and you’re experimenting. And then I went to take another job, and Willem took another job. And so when we restarted that endeavor, we said like, “So, I’ve seen now other companies and you’ve seen other companies. And the base, the base principles, they are always the base problems, they are always the same.” So we started translating those concepts into what we then started calling our IT/OT cooperation patterns, which we published already three years or so ago…

Willem van Lammeren: Years, yeah.

David Arens: …on our blog, became one of the most read, most shared content. Why? Because, and that’s the strength of DevOps, team topologies. It’s like it’s very visual. You, you’re drawing circles, and you’re explaining like, okay, you are an IT team, you’re an OT team. This is the type of problem you have. Are you looking for small type interactions? Are you looking for first steps, versus, and maybe we shouldn’t go over all the patterns because the book is, more than a third is on those patterns, versus the more complex projects, where you have a higher change velocity, where you have higher dependency on each other. And that then became originally eight, but we boiled it down to seven patterns from very simple ones to very advanced ones.

And I think the most important, yeah, main idea almost we took is that we believe at this point in time for most companies, we are too early to really break the silos. So it’s too early to say like, “You know what? We’re gonna remove the OT function from operations and we’re gonna move it into IT, or we’re gonna remove IT and all, IT in its entirety and move that to oper- to move that to operations.” That’s, it’s too early. It’s probably not gonna work in most environments. So this is where the patterns, they acknowledge the silos, but they give you a language to cooperate given those silos and make those silos less, yeah, less of a problem. And then hopefully after a while, you, you don’t feel as being in a silo any longer. You feel like we are just colleagues amongst each other trying to collaborate on a, yeah, on a digital project.

Willem van Lammeren: I also think silos are not always necessarily bad. I mean, if you work in a company and I want a laptop, I want a laptop. I don’t want to start a multidisciplinary team that comes together to examine how we can… I just want this thing fixed, and it’s a contract, and next day it’s there. So silos are great, but we’re adding a bit more nuance to the concept of cooperation within this context of IT and OT. Again, based a lot of on team topologies. And we take them, like David said, from observations in the field from talking to hundreds of people in manufacturing. We distilled ways that you can break down the silos. And it’s not always make a product team. There’s like steps in between and each step has its place.

[00:22:41] What Are Some Industry War Stories That Highlight the Challenges Between IT and OT?

Henry Suryawirawan: Yeah, so before we actually dive deeper into the collaboration, the cooperation patterns that you mentioned, right? Because very interesting that you mentioned about these silos, but for us maybe to kind of like relate a little bit, since you have been working in the industries for quite some time, maybe seeing a few different companies, talking to different peoples, are there kind of like war stories that can kind of like illustrate this challenge between IT and OT?

David Arens: War stories. Oh, so…

Willem van Lammeren: I have the one from the book where we open the book with.

David Arens: Oh yeah, why not?

Willem van Lammeren: And it’s a true story.

David Arens: Yeah, I remember that one.

Willem van Lammeren: It’s a true story. So factories make stuff. Stuff needs to leave the factory, otherwise you have a problem. Very simple. Trucks come in, trucks come out. Now, there’s regulations. Trucks cannot leave without the correct papers and correct paperwork and all, everything comes on top. So I get a phone call around 8-9 o’clock, and they’re telling me, “Willem, the trucks are not getting their papers.” And we had just automated everything. I mean, everything was automatic. They would scan stuff, things would come out. I log in remotely, I see everything is generated and like, I fucking hate printers. Why don’t they fucking print this thing? Yeah. If the trucks cannot leave or enter the site, David, at the size of that site, we would have like huge, huge traffic jams. They would be on the news. I mean, it’s like hundreds of trucks blocking the highway. So off I go to the site. People from logistics were there already on site, and we’re like, “Why the hell does this thing not want to work?”

So we go through everything, every possible little piece in the pipeline that could fail or not. Does the printer work? Yes, the printer does work. Does the software to print work? Yes, it works. The servers, everything works. We start a major incident, so we get everybody on board to find out what’s happening. And I think it’s like 2:00, and we’re like, “Okay, did anything change in our ERP system? Anything in the world? Like whatever. We have no clue. Anything.” It was like 2:00 in the morning at that time. And they said, “Yeah, we did a small patch in China.” It’s like, “Okay, but we’re on the other side. Whatever. Can we reverse that change?” Because we’re like completely out of options by then. Everything should work. It doesn’t work. They reverse that change. By then it’s 2:30, 3:00 in the morning, and it suddenly all works again. So a small change in a form for Chinese customs changed some symbol, which meant that on the other side of the world, trucks could not leave.

David Arens: And that has a physical impact because, and it’s, I would say this is a, this could be any IT incidents, Henry. So that it’s like, it’s not that our IT is different from like your IT, right? The big difference here is that if these trucks can’t move in a certain pace, silos on the plant itself starts filling up. And we’re, in this case, we were talking about hundreds of trucks a day. So silos would start filling up, and at a certain point in time, silos would be full, meaning that you have to shut down production. So this is where this physical link suddenly becomes extremely important, where we often only have a very limited amount of time to either fix the problem or implement a workaround. And that’s the problem also we know from IT, is the more we start automating, which we should, the harder it becomes to fall back to a manual workaround. Because you need to have the knowledge to do for the workaround, you need to have the procedures, you need to have people to be able to execute upon the workaround.

So it simply becomes more and more and more tricky to do so while there still is that physical impact. And in, I think in Willem’s story, and are many, many, many others, it’s always the, I think the big difference here is, problems occur, yes. But once you come together in a team of people who have one goal, and that is to fix the problem across the silos, then we are always able to do that. But it becomes so much easier as if the other side understands that physical impact. And that’s hard when there is, for example, distance or IT is outsourced to another country, et cetera, et cetera.

Willem van Lammeren: I had to send pictures. They said like, “Is this really a major incident?” I’m like, “I’m gonna take a picture of the trucks waiting. Like I’ll show you the traffic jam outside and then maybe you’ll believe me.” And then they believed me. Another really good example that a lot of people in our industry will recognize, David, is the PC you cannot patch.

David Arens: Yeah.

Willem van Lammeren: Or the Windows 95 PC.

David Arens: Like Windows, yeah, whatever old version, Windows XP, also 95, but Windows XP is still a very, very common operating system to be found on factory floors. Why? Because the thing was bought at a certain point in time as part of a very expensive, very complex, very highly…

Willem van Lammeren: Multimillion. Huge.

David Arens: customized asset, and it’s sitting there. And it’s, and it’ll keep on sitting there. And we have examples of controllers being decades old, but we still need to support them. And that’s also really interesting. Like, how do you deal with that? How do you inter- how do you make sure that from a cybersecurity point of view, yes, of course, Windows XP is quite cumbersome, to put it that way. But still we need to be able to be, we need to deal with that. So there are ways of course to secure it. There are ways to connect to that type of machines. It is something which is again, also really typical for our world.

Willem van Lammeren: But also very typical response of IT is like, “That’s not allowed. You should now get a new PC.” And if that’s the approach you’re gonna take to the part of the company that’s actually making the money and say like, “You should get a new OS, ask your vendor.” They’re like, “From what planet do you come? This vendor doesn’t exist anymore,” or whatever. They don’t care about… They’ll say, “You want a new Windows version? Sure, buy a new pub.”

David Arens: And I had that point. I had that discussion multiple times, obviously. And at a certain point in time, again, yes. So this is like, this is not a red security risk anymore. This is a black security risk, like the worst you can get. So IT comes in, tells operations, “You will have to shut it down.” And then operations said like, “Okay, fine. Like if you pay for the new machine, we’ll do so.” Okay, we’re gonna do that. So how much would that investment be? So many million euro. Why? Because it’s not about replacing the machine. It’s about the entirety of the assets. Sometimes with customized hardware, all you… And I know that that was a discussion where IT said, “But that’s our running budget for the next year.” And operations said like, “Yeah, sure. If you want to replace the machine with your entire running operation, or your running budget, please do so. We’re all good with that, but we don’t have it budgeted, and we will do so in the next turnaround,” which might be a turnaround means where if you shut down the plant and you re- yeah, you replace some, you do critical maintenance and you replace critical equipment, which might be in five or six or seven years.

Henry Suryawirawan: Wow, this is such a fascinating problem that you’re dealing with, right? So I could imagine, you know, working with legacy technologies, right, that probably is already out of support. It’s end of life, right? And probably, yeah,

Willem van Lammeren: Beyond end of life.

[00:30:57] What Collaboration Patterns Can Help Resolve Tension Between IT and OT?

Henry Suryawirawan: and some security vulnerabilities that potentially could be there, right? But definitely this is quite a challenge, right?

So let’s probably dive deep into some of the collaboration patterns that you mentioned, right? Maybe some of the common ones that you think, are, you know, probably can solve some of these challenges, right? When IT and OT works together rather than, you know, always having this tension because we don’t understand each other better.

Willem van Lammeren: Well, David, I’m gonna start with maybe the more controversial ones because everybody knows, well, let’s do cross-functional product teams, let’s put them together and it’s gonna be great. I mean, sure, you’re gonna do that for the high value, high visibility stuff, but there’s like tons of other work that needs to happen. David and I say that good shadow IT is actually a good thing.

David Arens: Yeah, please don’t, please don’t shoot us. It’s a shadow IT. It’s actually one of my favorites. But I’m gonna explain it here. So Willem already mentioned that we are working here with two technical functions, highly technical functions. OT people are very, very used to work with servers and networks, and patching and security, and data transfers and all the good stuff. So yes, they might use other software or they might use other systems or other ways of thinking, but they are very used to that world. So for them to buy one additional server in a server park of a couple of hundreds , or more, it’s no big deal.

So, we actually say that if there is no cooperation possible because the silos are just too far away from each other, let’s just start with some simple, good shadow IT, for example, where… You can also have it the other way, a shadow OT, but that’s, let’s keep it with shadow IT, where these people, where experienced OT people, they install an extra application, et cetera, et cetera.

Why is that good, Henry? It’s good because they learn new skills. They learn what it takes to administer a database or to, I don’t know, integrate in some middleware application or…

Willem van Lammeren: To manage the application lifecycle, to do support, to do security work. I think good shadow IT also means doing it within the correct boundaries. So it doesn’t mean you give admin rights to everybody and install what the hell you want and just open the firewall to the outside world. It is doing good enough IT with a lower complexity, solving local problems.

David Arens: Which brings us automatically to facilitation because obviously that shadow OT is only gonna get you that far. There will always be a point where you need to open it to the corporate network, where you want cloud connectivity, where you have a vendor sitting on the internet somewhere who wants some data, you know, et cetera, et cetera.

Where the CISO comes in and says like, “We have to do a security audit.” You know, all the good stuff. So that brings us to facilitation, which is our next pattern. Facilitation means that we’re gonna work together. We are desig- we are, I would say selecting, designating an OT architect and an IT architect. We don’t ask them to change jobs. We don’t ask them to change, you know, reporting lines or whatever. You just have two assigned architects who have to come together and solve, I would say the rather smaller, easier problems, where someone, somebody needs access to something, a system needs access over the firewall to whatever. There is a valid security concern raised by such and such, and we need to solve that. Let’s just make sure that you have two colleague, two colleagues, two coworkers, both having the task to just fix those problems. Very simple pattern, extremely powerful. If that all works well, you know, if there is some shadow OT and they have some experience what it takes to do whatever they wanna do, if there is some facilitation where you know people, they come together, where the, both the CIO and COO understands like, you know, if we come together actually we can take steps, that’s when we can step into the more advanced, the more advanced models. And this is where you typically, you know, especially for the larger projects, the larger, yeah, requests, the larger projects, you would start a project team. But the project…

Willem van Lammeren: I think also it’s good to understand within manufacturing companies, everything is a project. I mean, it’s the way they become big. If you build a plant, I mean, it’s a project. It takes years. It’s in their DNA. It’s how stuff gets funded. It’s how stuff gets done. Getting real product teams more focusing on continuous improvement, it’s much harder. They’re really capex driven, and that’s something you need to take into account. So just going there and say, “Let’s make cross-functional product teams,” nice, but that’s not how funding happens in those manufacturing companies. So we did add the project team as one of the patterns and a great pattern to start. So if you have the focus, you have operations and IT focusing together on a specific mission, a cross-functional project team is definitely the way to go to get things started.

David Arens: Yeah. It’s a way to start, not a way to end. And it’s also something we describe in the book is that one of the biggest anti-patterns when starting a project is not discussing what’s gonna happen after the project. Because again, if you’re just in this capex world and you say like, you know, I’ve now allocated 500,000 euro or whatever to this project, you know we’re gonna execute it. In the beginning, typically everything goes really well. Like, yeah, two things can happen. So either the project team just dissolves at a certain point because people are, you know, pulled in other directions. And then your, the thing you delivered, like version one is there, but what’s gonna happen now in the years to come? It’s, there’s gonna be one big void. It’ a very common anti-pattern unfortunately. Or other problem is like, if you then go like, “Oh, we shouldn’t dissolve.” So now the project team becomes permanent by accident, and it actually becomes a third silo now. So now you have the IT silo, the OT silo, and then the project silo…

Willem van Lammeren: The innovation silo or…

David Arens: The Industry 4.0 silo, whatever you wanna call it. And it also happens quite often. So we say when you start a project, and that’s perfect because operations know how to do projects, you need to discuss the end in the beginning. Like we’re gonna start, but how is this gonna evolve? And this brings us, of course, then to the product team, which is really known in the IT world and where we believe there is a big value also in the OT or the IT/OT world to have those product teams, but not for everything, yeah.

Again, as Willem said, we don’t need a product team to order a laptop. The same is true for an average digitalization project where, you know, you can say like, “Okay, we have an existing OT team, and they can just, you know, they can deal with whatever is being built.” But there are pro- there are several products where we really want that permanent product team as well. And that’s, I think, the final pattern we describe in the book.

Willem van Lammeren: It’s a hard pattern to get to though. Again, using that manufacturing context in mind, a lot of money and effort goes into building that thing. And once it’s there, okay, you do the minimal maintenance. But it’s doing its function. Software is not like that. It evolves, it grows, and it moves much faster than, you know, than a physical asset does in a plant.

David Arens: And there are so many great books in the, yeah, in the world, but also in the IT Revolution family. And they all talk about the same type of problem. They talk about the fact that, you know, if we take years to describe the perfect digital product, chances are it will never be delivered. Because by the time we start building, you know, the idea, the specs, the things we want, they already changed. And that’s of course also a common pitfall we have in our world, but we need to marry, yeah, both sides a bit better. Otherwise, you know, these people from IT, they will see, they will be seen as aliens. So you need to be sure that those two ideas, those two ways of working, they come together.

[00:39:43] Is Tesla the Ultimate Reference Model for Modern Product Teams?

Henry Suryawirawan: Yeah. I can really see the progression that is required between, you know, the IT/OT, right? And especially I think it might differ based on the kind of like types of industries, right? Maybe when mentioning about, you know, water plants versus maybe, you know, car parts manufacturing and all that, probably they kind of like have different complexity. But I’m also guessing like product team will be something– Like the one I’m really familiar with because I really have limited knowledge is like we always see Tesla, right? Producing Tesla. It’s such an advanced, you know, I don’t know how they do the machinery. They can push updates. They can even have a factory working by itself. So is it like the probably good reference as a product team? Or like what does Tesla do differently that make them such an advanced, you know, kind of like operation technology?

David Arens: Yeah, so, so Tesla is an IT company who happen to make physical assets. It’s the way they are wired. And I’m not saying physical wiring, but the way they are wired in their mind. And I had many interactions with Tesla employees. It’s extremely, extremely, extremely interesting on what they do. But still they come from this IT product first mindset and they try to apply, yeah, yeah, convert it into the manufacturing, the physical world. And it’s extremely interesting. Having said that, it is just, it’s impossible to change existing manufacturing companies. They’ve been around for… And that’s also their Achilles heel, obviously, yeah. But if you come from a, what we call a brownfield plant, so the majority, well, the majority, almost every manufacturing company already exists. The ones who are built new, that is actually, yeah, it’s, that’s a very, very, very small part. They exist. Changing their entire philosophy, mentality, way of working, way they approach their production lines, et cetera, et cetera, is almost impossible.

And it’s, there’s is an example from Tesla where they, I forgot the name of the town, but so they took over an existing production hall of a car company which went, yeah, stopped or went bankrupt, I don’t know. But they had to remove all the assets from the production line and start all over again because they weren’t able to apply, you know, their philosophy to the existing assets. And I think that’s probably the most important learning here is that it’s just a different, a different world. And then of course, there are also many companies who are just regulated. Like the regulation states, “You shall do it this way.”

Willem van Lammeren: Yeah. Pharmaceuticals is really known for that. If you want to change one small little thing, the amount of paperwork very often costs more than the benefit you would get.

David Arens: Yeah. Yeah.

[00:42:41] How Can OT Teams Adopt CI/CD Through a Data Platform Abstraction Layer?

Henry Suryawirawan: Right. I really hope people who listen to this conversation, really open up their perspective, right? Because we are so used to digital products. It’s so easy to build. You know, we can iterate, you know, CI/CD and deploy to production multiple times a day. But I think when looking at this world, right, things are not that kind of like smooth and always easy, right? Which, yep.

David Arens: But, but one of the things we really advocate for is we have to step into that way of working as well. We also need to do CI/CD, maybe not to the extent of 1,000 deployments a day, but to multiple a day. And the reason… Oh, no, not the reason, but one of the ways we can get there is making sure we have an abstraction between the physical assets and the physic- and the control of that physical assets. And then yeah, we refer to as a data platform. You could also say a middleware layer or whatever. But decouple that from the very, very fast changing IT world with something in between where you can add that speed, where you can add that flexibility, that new way of working, that, you know, config by code and all the good stuff. And then step by step, we also see, but that’s gonna take years and maybe even decades. Step by step, we also see that the way these control systems are being programmed because that’s also just, yeah, codes of course. We also see that very slowly, they’re also adapting to new principles in the IT world. And for example, using Git, using version control and that type of things, doing more config by code. It is something which we now also see very slowly though, but we see it also happening on the, on this lowest level.

[00:44:37] What Will Be the Real Impact of AI on Operational Technology?

Henry Suryawirawan: Yep, thanks for the addition, right? So I think, definitely very exciting, which I think one of the big question these days, especially happening in the IT world, is the AI part, right? Like we all seem to think AI will kind of like solve a lot of things, including building software. So my question for you, both of you who have been in this OT world, right? What do you think of AI? What was the impact really AI to your world actually? Is it gonna be the same, like replacing a lot of kind of like, you know, writing software, software developers and all that? Or is it something that is kind of like will take some time to actually get more traction?

David Arens: It already starts by defining what is AI because AI is more than generative AI. Most people now know it from, you know, I mean, we’re using natural language to communicate with whatever. But AI has been around in the industry for decades. And machine learning, is front and center in so many applications to optimize processes in vision systems, and you name it, in vibration monitoring, you know. So many examples where AI is already front and center. But I think what we now see is there is this new wave of, yeah, generative AI, and I do believe that there is gonna be an impact. It’s just gonna take some time, and it’s gonna be slower and more, yeah… May- slower is maybe the wrong word, but I think we, it’s the applications are going to be more defined than just, you know, we’re gonna try something and Willem, you’re trying it right now, and I don’t think you’re building a next chatbot to, you know, interact with the plant.

Willem van Lammeren: Well, I think there, there’s a couple of things. First of all, a very big factor that determines how AI is impacting coding is that when all the information that you have, need, use, your processes, everything is already digitized. It’s pipelines, it’s your repositories. David, I don’t know about you, but most plants are not stuffed with highly contextualized available data through APIs.

David Arens: So that–

Willem van Lammeren: The data is hidden. It’s hard to get to. Data quality is not always there. It’s inconsistent across plants. So I think first of all, there’s a big aspect of data availability that makes a big difference.

David Arens: Yes, and you need that data. Also, Henry, because like almost every plant is unique, which means that, it’s like you can’t start training a general model on some general internet scraped data from Reddit whatever you… So you do need that plant data.

Willem van Lammeren: Yeah. I always tell people in projects like AI can do cool stuff with or without data, but if you want to make it useful, you’ll need your own data to make it relevant. Even if you get that data part solved, there is still a lot of work that needs to happen to make that AI useful because the outputs that your AI gives when you’re coding is the artifact you need.

In manufacturing, the LLM is not gonna make me chocolates. I mean, so you need to find ways to integrate it into your processes to find out how it can improve those processes. So it’s not that…, the impact is not the same as it does for software engineering. I mean, in software engineering completely changed. Maintenance engineering, not yet. We need to get the data, and if it does, it’s gonna be more of a support to the process. And even then, getting the data is not going to be so easy. We might have sensor data, but we will also need transactional data. We will need world models. We will need to combine that knowledge. Nothing of that, what I’m saying is like impossible, but those are not artifacts that we have ready-made, ready to use, plug and play. We need to build those.

David Arens: Which means that I think if we lean upon what we see in the IT world today, what are the… When we, especially when we talk about GenAI obviously, what are the things we see today? That’s configuration assistance, yeah. So, making it, so what we know from the IT world, we see that more and more companies today, more and more software co- sorry, software companies and not the end users, but the software companies, they are obviously making their software easier to use. So instead of you programming whatever control loop, whatever data integration pipeline, whatever, yeah. Yes, obviously we see many companies now having some way to, with natural language, say like, I want to do this, and then the code just comes out. So that’s happening. It’s happening again at a little slower pace, and there is some difference between companies. But that’s a very big aid, especially because we actually don’t have enough engineers in the manufacturing world globally, yeah. So everything we can do to make their job faster is not gonna… I would say there won’t be any layoffs in our world anytime soon, because there is just a shortage in skilled people.

So that’s brilliant. And something which we also see now popping up is making it easier to search through documents, you know. Willem already mentioned pharmaceutical industry. We see, for example, in the pharmaceutical industry, generative AI being used to make the legal processes, so the creation of all these documents, around changes to make that easier. So those are things we do see. But you see it’s a bit more on the, you know, the document side of things. It’s a bit more on the end, yeah, the pure coding side of things, what you expect, to be honest. But its real impact on the process itself, that’s gonna take a while.

Willem van Lammeren: Yeah.

Henry Suryawirawan: Yeah. I think you mentioned about a very interesting example just now, right? Using GenAI to produce like maybe chocolate, right? So maybe, I don’t know, because of the unpredictability of AI output, right? Maybe sometimes you can get a, like a mix of chocolate, which makes it interesting.

Willem van Lammeren: But I dare you to ask, you can ask ChatGPT, “Please make me a chocolate.” You’ll get a nice piece of text explaining you how to make chocolate, but you still have to go out and make the chocolate yourself.

Henry Suryawirawan: Yeah, which brings to the as- the safety aspect, right? I don’t know also how safety it is, you know, using GenAI to produce the chocolate.

Willem van Lammeren: Hallucinations are also big, a big factor, especially in manufacturing when it comes to trust. Trust is really something that has to be earned, always with users. But engineers are like a bit difficult in terms of users. Like they want to know exactly why did you propose this and how did you come up with the solution and what data did you get. If you can do it, you’ll gain their trust. But especially with AI, if they just get a hint of any hallucination, you can quickly lose trust.

[00:51:54] What Is Driving the Shortage of Automation Engineers in Operations?

Henry Suryawirawan: Right. Also another interesting that David mentioned is about, you know, the lack of engineers in the OT world, right? Maybe one funny thing is like, maybe all the software engineers who feel threatened, maybe you can now explore the OT side.

David Arens: You’re welcome!

Henry Suryawirawan: But I think it’s very interesting. Why is it such a lack of, you know, software eng- like, what do you- do you call it software engineers? Like, why is it the lack of…

Willem van Lammeren: Automation engineers.

David Arens: We call it automation engineers. Like so it’s an education problem, Henry. It’s really an education problem. It’s not because there is no job certainty or it doesn’t pay well or, you know, whatever, or it’s not cool because it’s like, you know, how cool is it that you’re writing code and you see things move like up.

Willem van Lammeren: It’s like your Raspberry Pi project at home, but like at a massive scale.

David Arens: But it’s an education problem. So actually I had, but it’s already a while ago, two years or so ago actually, I had somebody from the US on our podcast talking about education as well in the US then. I can only talk about Europe. But automation engineering is something like you, there are exceptions, but in most universities or most high schools, that’s typically not really a path you can choose. Like computer science, that’s a path everywhere. You know? It, like, there is not a single educa- high school and university type of, which doesn’t get, yeah, where you can’t do something around, you know, AI or computer science. But automation engineering is typically more part of a technical educational program which is in my opinion, totally wrong.

Like that’s, it’s like, it should be more broadly available. But that’s the most important thing, is that you, the choices people make, and it’s often also seen, and what I’m gonna say right now is maybe, it’s, I mean it in the best possible way, but people often see going for a technical education as less interesting for their future career, right? Where, you know, they wanna be the, they wanna have a master’s degree in whatnot. Whereas these technical jobs are what, you know, our society is running on, on electricians and automation people and plumbers and, you know. And that’s a bit of the problem, yeah, there is that there is this split and there are, there is thus just not enough people coming from that education. And I know it’s a problem in Europe. I know it’s a problem in the US as well, although it is changing. And I, yeah, I don’t know about Asia, but that’s the most important one.

[00:54:51] Are Humanoid Robots Overhyped for Manufacturing?

Henry Suryawirawan: Yeah. How about robotics? I think we can see the trend is picking up, right? People also talk about robotics replacing humans and all that. So is it also something that you see picking up traction, you know, with this IT/OT work merging together?

Willem van Lammeren: Yeah, since the ’70s. Since the ’70s, I mean, robots have been in manufacturing since a long time. I think David recently you just went into a warouse, humongous, like one or two guys, they just walk around and they fix problems.

David Arens: Yeah. And then it’s… football pitches like in size, and you only have an operator who is dealing with the errors, basically. And everything else is purely, purely robots. But I think Willem, as you mentioned, for us, robot is something we’ve been done since decades. Like the fact that we are now talking about humanoids, honestly, that is not changing a- anything, right? Like humanoids, it’s very fun, but the manufacturing industry does not require humanoids. Let’s be really honest.

Willem van Lammeren: I think also most plants and, and processes are not necessarily made to fit very well to humanoids. Like I said, there’s already a lack of data in many cases just to get a normal AI model working, let alone having a humanoid walking in an environment where safety could be impacted. Now, are there no applications? I mean, stuff will come up, but automation is– has a lot to do with that part. Like just imagine, David, you need to engineer that pitches sized warehouse, fully automated, trucks come in, stuff gets unloaded, sorted everywhere. I mean, it’s a huge engineering puzzle that already exists today. It’s lights out. And it’s not just because of humanoid robots which get all the attention now.

David Arens: No, because a humanoid is basically a generalistic robot. So the, a humanoid is where for some reason we thought it was an idea to mimic how a human looks like in a robot form, which means that they might be, the, it might be possible to make them do generalistic tasks at a certain pace, we want that, we really want that to turn around. We need extremely performant robots to do one or a set of specific tasks very well, clearly defined. And we also see that changing obviously, yeah. So the robot world is changing a lot. We see more applications. We see, for example, programming robots becomes much, much, much easier today than it was some years ago, yeah. Generative AI, again, is a very, very big help. But having said that, like in my opinion, it’s an evolution. It’s already started, and it will keep on going. It’s not really like this big revolution where all of a sudden, I always say the joke I sometimes make is when I see this robo-dog walking around somewhere. It’s not because you buy an expensive robo-dog who walks around in your plant, that all of a sudden your plant becomes more optimal. Like, b- b- b- it’s a fun… It could make sense, yeah. It could make sense, for example, to have a robo-dog walk around a very hazardous place, where you don’t wanna send people, and there are brilliant examples for that. But that’s not gonna change the output dramatically of our, or even at all, of our manufacturing process.

Willem van Lammeren: Another thing to keep in mind is the amount of manual labor in most big industries is actually already quite low. Most of the cost is actually in the assets. I mean, just drive by a plant, just think about all the machines in there, etc. The IP that they have, the real manual labor is most of the time a fraction of the cost. So the business case to, let’s say, I’m gonna replace people with robots to save on labor cost, I think if you’re gonna look at the total picture, you would need to replace that human with robots. It’s gonna be a hard project to sell for now.

David Arens: And again, we’re talking about the manufacturing plants, the process plants, et cetera. Obviously in the supply chain world, that’s different.

Henry Suryawirawan: Yeah. So I think the keyword I picked is the generalist aspect, right? So we are seeing a lot of buzz and hype, you know, talking about, you know, more generalist robots where maybe I can see like robots folding your clothes, you know, sending delivery, you know, supply chain, you know, driverless cars and all that. So definitely it’s a new, probably a new kind of a industrial thing that maybe one day can be more interesting, you know. But obviously what you shared today, it, I hope it opened up the kind of like perspective for many of us that, you know, there’s this world as well, right, that we have to collaborate. That’s the first thing, right? Because you come up with seven different collaboration patterns, right? And how we can optimize working together. And also not to mention that we have to empathize with each other, right? So like IT might have its own practice, best practices, but at the same time, OT also has its own best practices.

I think we have covered quite a fair bit of topics. Is there anything that you think might be worth to also mention here before we wrap up for our last question?

David Arens: Buy the book.

Henry Suryawirawan: Nice one.

Willem van Lammeren: Yeah, I think the book gave us a chance to like really dig deeper on the topics we discussed here. Like really go a bit and take our time to find out what’s really behind those things. How do those cooperation patterns look like? How do you get started? How do you decouple the two worlds so you can start to move at different speeds? I think that’s all things you can find in our book.

David Arens: You see, and that’s also how our, we, how we work together. So I just give the stupid answers and then Willem, you know, he smooths things out.

Willem van Lammeren: Actually, I kill half of his ideas, if not more.

Henry Suryawirawan: But maybe that’s why the, you know, the collaboration actually works.

David Arens: Yeah.

Henry Suryawirawan: So yeah, thank you so much for sharing this. I think, for people who are interested, maybe you live in both of these worlds or maybe one world who are interested, you know, on the other world, please check out the book, right, it’s published under IT Revolution, right? So definitely, I mean, I assume by, picked up by IT Revolution means this book potentially can change the industry. Because we have seen so many examples where books from IT Revolution change the industry, like Team Topologies you mentioned, Phoenix Project, right, DevOps Handbook, so many of those, right?

[01:01:52] 3 Tech Lead Wisdom

Henry Suryawirawan: So I have one last question for both of you, right? I call this the three technical leadership wisdom. It’s like a tradition in my podcast to ask my guests to share about this so that listeners can learn from both of you. So maybe I’ll leave it up to you how you wanna structure it, but if you can leave some wisdom for us to learn, I think that would be great.

David Arens: Who goes first, Willem? Me or you?

Willem van Lammeren: You go first.

David Arens: So, my first one is I… So the title is, Be Like James. And w- so, in this case, I mean James Bond 007. So, you know, so for some reason I love the movie, the movies. So for some reason, Henry, like James Bond always has a dinner or lunch with his enemy, which like is super strange in my opinion. But when we are talking here about, you know, the IT/OT divide, unfortunately, tensions often rise in those discussions, and unfortunately we often treat each other as the enemy. So my advice is like, be like James. Step over the differences and have lunch or have dinner with your enemy. But then just doesn’t, don’t kill them, you know? Just start working on a collaboration model. So mine is Be Like James. Willem, which one did you bring?

Willem van Lammeren: And that is actually true. He is like James in that aspect. Mine is know how to be a gardener and a carpenter and when to switch. So we’ve managed decently sized teams and groups. And there’s two ways to do work. The first one is like the carpenter. You go in, you try to fix things, which is great, except it’s absolutely horrendous when you need to scale. You can only solve that many problems. So instead, try to be a gardener and look at your teams like your garden, and let them grow. And you need to find out what they need to grow, and that means working on their context, finding out what makes them work. And if all goes well, David, you can just sit back and relax just long enough to enjoy the garden.

David Arens: No, no, but that’s a really good one. And let’s finish with a, with just maybe with a more normal one, more normal one, Henry. I think the most important like takeaway when we’re talking about, yeah, projects, cooperation, et cetera, et cetera, is always like in, as I already said, always know when you’re starting a project, when you’re starting a new idea, understand what is gonna happen after the project. And that means that when you are in a meeting, when you are a manager or when you are an individual contributor, and you feel that everybody is talking about big ideas and, “Oh, we’re gonna do this and we’re gonna do that,” be that one who says, “Ho, stop. Stop for a moment. Like this is a brilliant idea, but how are we going to act? How are we going to, yeah, behave as a group, as a company? How are we going to be organized? How are we going to be budgeted after this brilliant idea is, yeah, is live?”

Henry Suryawirawan: Wow. I think this is very unique, right? First is be like James, James Bond. Second is be a gardener rather than a carpenter. The third I would summarize like be like an oracle, you know, kind of like predict what will happen next.

David Arens: I’m gonna remember that one. I’m gonna use a Be an Oracle.

Willem van Lammeren: It sounds better than be that annoying guy who asks the annoying questions during the steerco.

David Arens: Be like a gardener, be like James, and be an oracle. Willem, I’m gonna write it down in my mind and in the next podcast, those are the three.

Henry Suryawirawan: Right. So really appreciate your sharing today. I really learned, you know, new things, new paradigm. So not only just software engineering, you know, IT world, digital products and all that. So I really appreciate your sharing today. And hopefully, yeah, people can learn a thing or two as well from your world. And also, for example, like put safety in front, right? Rather than always, you know, releasing fast, break things, right? So I think we have to understand and empathize different world. And thanks so much as well for, you know, sharing the impact of AI on your world, right? So probably it’s also something that kind of like refreshing because all in the news that we see is like, you know, we are doomed, right? Software engineers is gonna be gone and all that. We’re gonna be replaced.

David Arens: But it’s okay. We’re gonna be fine.

Willem van Lammeren: There’s more engineers.

Henry Suryawirawan: There’s more eng- there’s more engineers, yeah. So, okay. So yeah, thank you so much. I hope, you know, many people will read the book, and I hope it can also change the industry just like any other IT Revolution books as well.

David Arens: Thank you so much.

Willem van Lammeren: Thank you.

– End –