#67 - Continuous Architecture (Part 1) - Principles and Essential Activities - Murat Erder

[00:01:09] Episode Introduction

[00:01:09] Henry Suryawirawan: Hello again to all of you, my friends and listeners. Welcome to another new episode of the Tech Lead Journal podcast. Thank you for tuning in and listening to this episode. If you’re new to Tech Lead Journal, I invite you to subscribe and follow the show on your podcast app and our growing social media communities on LinkedIn, Twitter, and Instagram. And if you have been regularly listening and enjoying this podcast, and love the type of content that I’m producing, will you support the show by subscribing as a patron at techleadjournal.dev/patron, and support my journey to continue producing great episode every week.

My guest for today’s episode is Murat Erder. Murat is the co-author of “Continuous Architecture in Practice” and the CTO of People and Procurement at Deutsche Bank. This episode is the first of a three-part series of “Continuous Architecture” episodes. So make sure that you check out all the other two episodes that will be released completely in the next one month or so.

To kickstart the discussion around this topic, in this episode, Murat started by explaining what software architecture is. And he provided a key insight that the architecture of a system is always there, regardless whether we plan for it consciously or unconsciously. Murat then explained in depth the six principles of continuous architecture mindset. Make sure that you listen to all of the six principles, which I think are really crucial to adopt this mindset, versus the traditional architecture that is generally cast in stone.

After the six principles, Murat continued by outlining the four essential activities for doing architecture that involve architectural decisions, technical debt, quality attributes, and feedback loops. Towards the end, we discussed the importance of data as an architectural concern. We touched on a few recent key data technology trends that impact and drive software architecture, such as NoSQL, eventual consistency, and data mesh. And lastly, Murat also emphasized the importance of designing data model properly as a prerequisite for a successful software architecture.

I really enjoyed my conversation with Murat, learning the essence of the continuous architecture mindset, the six principles, and the essential activities. And importantly, why software architecture is really important to define the quality that could make or break your system. And if you also enjoy this episode, I encourage you to share it to someone you know who would also benefit from it. You can also leave a rating or review on your podcast app or share some comments on the social media on what you enjoy from listening to this episode. I always love reading your reviews and comments, and they are one of the best ways to help me spread this podcast to more people. And it is my hope that they can also benefit from this podcast. Let’s get our episode started right after our sponsor message.

[00:04:38] Introduction

[00:04:38] Henry Suryawirawan: Hello, everyone. Welcome back to a new episode of the Tech Lead Journal podcast. Today, I have a special guest with me. His name is Murat Erder. He’s currently the CTO in Deutsche Bank. He’s based in London. Murat has around 25 years of working experience in variety of roles, pretty experienced from software development, architect, management consultant, and now within the banks. And interestingly, Murat has authored two books with almost similar title. First is called “Continuous Architecture: Sustainable Architecture in Agile and Cloud-Centric World”. And the second edition, probably I would say, is called “Continuous Architecture in Practice: Software Architecture in the Age of Agility and DevOps”. So today we are going to cover a lot about continuous architecture. Really looking forward for this conversation. So welcome to the show, Murat.

[00:05:24] Murat Erder: Thanks a lot, Henry. I’m looking forward to the conversation.

[00:05:28] Career Journey

[00:05:28] Henry Suryawirawan: So Murat, maybe in the beginning, if you can introduce yourself for people who may not know you yet, maybe highlighting your background or maybe highlighting your journey so far in your career.

[00:05:37] Murat Erder: Yeah, I’m happy to do so, Henry. And first, a clarification about my current role in Deutsche Bank. I’m the CTO for a particular area, which is called People Procurement and Legal. So we cover all the foundational technologies that the bank uses in terms of managing the people HR side, legal applications, corporate communications, corporate real estate, procurement, corporate security, etc. So I cover that area and not the entire bank. And I have been with Deustche Bank for 12 years and had multiple roles within the bank.

I’ll provide a brief background first. So I’m originally from Turkey, and then I went to the United States to do my Master’s degree in Electrical Engineering, and that was in University of Massachusetts. And then I moved to California to San Francisco, Silicon Valley. I originally wanted to work in image processing and work on actually core engineering around image processing, which was what my masters was about. But unfortunately, I couldn’t find a job in there, so I ended up going into software development, which as a core engineer in those days, and I’m talking a while ago, I was like “Software? Everyone can write software. Engineering is what’s difficult.” But I think it was a good thing that happened, because it opened a very wide world for me. Software and software development, of course, has over the last, you know, 25-30 years has expanded quite a lot. It’s much more prevalent in our current world than it was then. And it has given me opportunities to work in San Francisco, New York City, and now in London. So if it wasn’t for that, I might not have been able to have such a career of various locations and employers.

So after working as a software developer for several years, I got involved in the object-oriented analysis and design world, and that’s the same time I moved to New York City. I worked for two companies that were quite important in that era. One was Objectory that was founded by Ivar Jacobson, who came out with the term of use cases and lots of core ideas behind object-oriented analysis and design. And then that was acquired by Rational. That was when the Unified Modeling Language, or UML as we know it, was formed. So I was there during those years, as part of the company that developed that. So the way I think about software. And the way I think about architecture, that was very foundational of those years. I find that as a very good part of my career.

From then, I moved into management consultancies. I’ve worked for the traditional companies like KPMG consulting and Deloitte. In all those companies, I was part of architecture practice. I was focused on financial services in general. The first co-author of the first book, “Continuous Architecture”, Pierre Pureur, we worked together in New York City and KPMG consulting, and we collectively penned the architecture methodology for the firm at that time. Then I moved to London for personal reasons. After a period, I started working at Deutsche Bank, and I’ve had several different roles in the bank. I’ve been part of architecture group. I actually, for 3.5 years, I was accountable for running all our application integration services, like the core messaging platforms we have, which was a very good learning experience. I was part of the Chief Data Office when it was first formed, and I’ve been in my current role for the last four years. So that’s kind of a background about myself.

[00:08:45] Henry Suryawirawan: Thanks for sharing your story. I think it’s really interesting. And I can see now clearly why you are writing this book. First of all, you did it as part of the work, right? So continuous architecture is probably a culmination of all the experience that you have seen in your career and different roles that you have taken as well. And it’s really interesting that you worked in a company with Ivar Jacobson and also Rational UML model. I can still remember those things, although I rarely see it now these days for some reasons. So maybe we can also discuss about that.

[00:09:12] Software Architecture

[00:09:12] Henry Suryawirawan: But first of all, maybe if I’m being curious here, architecture is something that is kind of like words that is not commonly associated with, you know, like when you want to build something, you have to go with the architecture first. But some people actually say we should not go with architecture first. But in the first place, what is actually architecture? Is it so important for us to actually start with an architecture in mind?

[00:09:33] Murat Erder: Yeah. So that’s a very good question. What is software architecture? My core belief is that architecture is not something separate to the system, right? It is the system. As I think, Bass Clements and Kazman in their book say, “every software system has a software architecture”. In our first book, we called that the realized architecture, which basically says, as an architect, when you think about the software product, your main focus is to implement what’s running in production. That is the architecture. So your architecture is always there. It always exists. It’s not something separate. It’s not a piece of paper or it’s not a diagram. Though they can be very useful to communicate the architecture. The architecture is always there.

So when you say, should we focus on architecture or not? It’s really how conscious you are about the decisions you make to influence what’s running in production. And if you look at more formal definitions, like the IEEE definition of architecture, it really is about defining components, their responsibilities, and their relationships. So that’s what a more formal definition is. But my view is your job is to influence what’s running in production, and to make sure you make the right decisions so that you have a sustainable product.

[00:10:42] Henry Suryawirawan: It’s very interesting definition. I actually haven’t heard about this. It’s actually to decide what you are going to implement for systems that is running in production. I really love this definition because sometimes we think architecture is like a high-level thing. So you come up with a big design up front, you draw component diagrams, but essentially whether it gets translated to production or not, we’re not pretty sure. So when you say about having this architecture implemented in production, I think this is very crucial for me because so many people are probably not conscious about how the system gets implemented. Which is why earlier I said, I haven’t seen a lot of UML diagrams, a lot of architecture diagram. So maybe you can comment a little bit from here, from your experience recently, is this still a case that people writing, big upfront design architecture, or is it something that is already changing a lot?

[00:11:28] Murat Erder: I think it is changing a lot. And that’s why we wrote our books as well. To give a little more context on the two books. The first one, “Continuous Architecture” was written in 2016. And then the recent one just came out this year, which is “Continuous Architecture in Practice”. The premise of those books was to bridge that gap, right? Because there is that traditional view of architecture being big design upfront, separate groups, separate teams. You know, especially in large corporations, you have these teams that would come and review what you’ve done once in a while and tell you’re wrong, and things like that. And that is, I guess, the negative vision of architecture. Then you have the more Agile Extreme Programming and just Continuous Delivery world where you just have the teams under the same. We have to implement, we have to implement. We don’t need an architecture. We’re trying to bridge that gap. In the example we gave is you need to build a cathedral. So the traditional architect says, where is the five-year plan? While the Agile developer says, where’s the shovel? What we want to do is bridge that gap. We did construct a series of principles and approach, and really thoughts on how to do that. Because we believe architecture is important because it’s about making sure you have a sustainable product.

[00:12:42] 6 Principles of Continuous Architecture

[00:12:42] Henry Suryawirawan: So I like the analogy of building cathedral. So if I can ask you personally, right? Just now you said the traditional architect will ask the five-year plan, while the Agilist will probably ask, give me a shovel. What will be your answer in this case?

[00:12:54] Murat Erder: I would say we have to, it’s a little bit of both. It’s really around the six principles we have. One thing to clarify, continuous architecture is not an architectural methodology. It’s really a mindset, and almost a way of working, a way of thinking. So what do our six principles say? The first one is you should architect products. Lots of people think about projects on what I need to implement, but you should think about what does my software product I’m implementing? And what is its journey? So in the cathedral example, what is the cathedral? What’s its purpose? What’s my MVP? If you could have an MVP for a cathedral, maybe it’s not the best example. But you know, think about your products is the first one.

The second one is focused on quality attributes and not on functional requirements. We need the non-functional requirements of a system: security, scalability, resilience, performance, etc. Because those are the key things that will really test your system or your architecture. So really focus on those, which we tend not to as much, at least formally. And we only sometimes deal with them when we have big issues.

The third principle is delay design decisions until they are absolutely necessary. First of all, the core unit of architecture is a design decision. That’s a core belief I have, and you have to make the decisions at the right time. That’s a principle we took from the Agile world. Again, you can’t do a big design upfront. You have to evolve and make the key decisions early on, but delay certain decisions till a later point. Because if you don’t do that, then you end up over-engineering your system, and just build something, try it out, etc, move on.

Principle four is architect for change, leverage the power of small. So that’s the loosely coupled internally coherent components that’s been around for a long time. Now, obviously everyone is talking microservices. But that principle is really around that, is think about where your architect will change, have independent components that are loosely coupled.

Principle five, which is again, five years ago, sounded more revolutionary than it does today, is architect for build, test, deploy, and operate. So when you’re architecting the system, don’t think about just, as I said, the realized architecture, but also you think about how you build it, how you test it, how you deploy it, and how you operate it. So the architecture should cover all of those aspects.

And principle six is acknowledgement of the Conway’s Law, which is model organization of your teams after the design of the system. And what that means is there’re two aspects of that. One is, again, back to the Agile world, we truly believe in cross-functional teams. So don’t have separate database team that’s separate than an app development team, that’s separate than a testing team, right? We want to avoid that internally coherent things. But also, there’s a more macro interpretation of that is, acknowledge how you are within the organization, and what organizational constructs you have. Because Conway’s Law says that each software system or each system will reflect the communication structures of the organization that build it. So there is that natural way how organization structure does influence the architecture of the systems that support that organization. And you have to acknowledge that. Either use it as a constraint and you deal with it because you know that’s just how it is. Or you try to fight it if that’s the right fight to go after.

But those are the six principles, right? So that would be our answer is, think about architecture with those principles in mind. If you apply them consistently, then you’ll find that middle of the road between the traditional architect and the developers.

[00:16:21] Henry Suryawirawan: So I love all these principles because it seems to cover a lot of trends these days in technology, right? Microservices, architect for change, leverage power of small, Conway’s Law. A lot of people refer to team topologies, what they call stream aligned team. Also architect for products instead of projects. I think people these days are talking about product oriented mindset, instead of project management. So all these seem to be the principles of continuous architecture. But in the first place you said, this is not a methodology. So what is actually continuous architecture? How do you classify it? Is it something that you always have to architect your system every sprint? Something like that?

[00:16:54] Murat Erder: No, No. It’s not. As I said, it’s an approach, and it’s a way of thinking. So what we’ve always said two things. We’ve always said we have our principles and then we have a toolbox of techniques and recommendations we make. So in the first book, I guess our toolbox was more theoretical, in the sense that it included things like, examples could be how to use sequence diagrams could be one, or value chains to understand your business, or there was a technique like QFD for dealing with mapping requirements at different levels. In our newer book, which is more continuous architecture practice, which I think is, will be much more valuable for a much larger user base. It’s all around architectural tactics. What are the architectural tactics of dealing with security with resilience? What are the architectural decisions you need to make? So, in essence, that’s what continuous architecture is. It’s a kind of a belief in those principles, and keeping those principles in mind. Explicitly make an architectural decision to meet and apply architectural tactics for the software product you have.

[00:17:56] Focus on Quality Attributes

[00:17:56] Henry Suryawirawan: So I also have an interest in your second principle, which is focus on quality attributes, not on functional requirements. Sometimes, I mean, if we all have built products before, maybe mostly the product managers, the engineers, they all focus on functional requirements or the software features. So in this principle, you are saying that we should focus on quality attributes, not on functional requirements. Can you actually explain more about this?

[00:18:21] Murat Erder: Yes. And actually, if you look at our second book or “Continuous Architecture in Practice”, it’s probably embodiment of that principle in the book. The view is that you can probably architect the system in multiple ways to meet the functional requirements, but it’s really your quality attributes that will make or break your system. If your system cannot scale, if your system cannot perform, or if you have security breaches, all those things is what really makes or breaks their system. We tend to definitely not understand those requirements well, or we’d never, in general, are very disciplined about capturing them. We sometimes ignore the architectural decisions, not always, but the architectural decisions that impact them. So one of our core things we write in the second book is about the feedback loops and building the appropriate test harnesses. That’s very important. When you build a system, you should build a test harnesses for not just your functional requirements, but for your quality attributes as well.

[00:19:16] Essential Activities

[00:19:16] Henry Suryawirawan: Which is actually a good segue to what you cover in the book called essential activities. You have covered what are the essential activities that we should do in terms of being a good practice of continuous architecture. So there are four essential activities that you mentioned. Maybe you can elaborate each of these activities so that we are all aware of how to actually do this continuous architecture properly.

[00:19:37] Murat Erder: Yeah. And actually those essential activities were not in the first book. They came in the second book. And it’s almost to address the same question as you said, what is continuous architecture? What do you mean by doing good architecture? It was kind of also done from our experience of the three of us implementing multiple systems. Essential activities that if you do nothing else, you have to do these four things is what we’re saying. You can draw as many diagrams as you want, as few diagrams as you want. You can set up your teams like this, like that, etc. But these are the four things to say, I’m doing an architecture.

There are those four components, which are architectural decisions, technical debt, quality attributes, and feedback loops, and I’ll try to verbalize it a little bit. The main thing is architectural decisions. That’s your core unit of work of an architect. You know the decisions you’re making and the decisions you’re not making and their impact. That’s very important.

The second thing is you have to understand your technical debt. What is my technical debt that I have on my current platform? And when I make a decision, am I reducing or adding to that technical debt. Because that’s your long-term view of how sustainable the product is.

The third thing is quality attributes, which we just spoke about. You have to focus on quality attributes. You have to make sure you understand your requirements from, as I said, scalability, resilience, etc, and you have to build your test harnesses around those things and just get a view on how well you’re meeting them.

Finally, it’s implement feedback loops. That’s the most important thing. If you look at Continuous Delivery, Agile, all those things are about quick feedback loops. We deliver something in a sprint, two sprints. Are the users happy? Are they not happy? That’s a feedback loop. I have a test harness for scalability. I run that. Do I get a feedback loop? I make an architectural decision that I’ll have two components rather than one. Is that working? Is that not working? That’s another feedback loop. The way you set up your team and the way you develop, you should be getting feedback loops on all those aspects as quickly as you can. There’s “Evolutionary Architecture”. It’s a book that was written, and that has a very good example of like fitness functions that they use, which is another way of feedback loops. So that’s the essential activities. Know your decisions, know your debt, understand your quality attributes and get feedback as frequently and quickly as you can.

[00:21:55] Architectural Decisions

[00:21:55] Henry Suryawirawan: So if we can go cover them slightly deeper, a little bit. In the first place about architecture decisions, I like when you mentioned explicitly just now, it’s the things that you are making decisions on, and the things that you are not making decisions on. And the emphasis here is the things that you do not want to make decisions. Maybe you can explain why is it important to actually explicitly mention the things that you are not making decisions on? Or the things that you say, “ah, this is what I don’t want to choose,” for example.

[00:22:22] Murat Erder: It’s a good one. In our second book, actually, one thing that I hope users, readers, not users. That shows how much I’ve been in software development. Readers would find useful is, we have a case study on the book around that case, which is a trade finance system. It’s not an actual system we implemented, so it is fictional and it is vague in certain areas on purpose. But we have listed all the architectural decisions we made. “Do we want to use asynchronous communication” is an architectural decision you can make for meeting the scalability requirements?

The way we approached it in that book is we said, “Well, it’s doing request reply, RESTful interfaces, much more simple to implement and get your product out rather than building asynchronous mechanism. So we’ll go with that first, and we’ll bridge that gap when we understand if we have such requirements, and if we have to meet those requirements.” So we’re not suddenly making a decision upfront saying, “Oh, this system has to be scalable, so everything has to be asynchronous or streaming or this or that.” We’re saying we’ll go with the simple first, and we’ll delay that decision later on. There are certain decisions which you can’t delay obviously, and the core ones are about decisions around your data architecture, and how you’re dealing with the database technologies, because they can be much more invasive.

Another example of again, rules engines is another one like, I’ve seen lots of systems built where people say, “Oh, we will have lots of changes in our rules and lots of flexibility required. So let’s just put a rules engine in the core of the system.” But you don’t really know if you need it or not. So again, that’s about delaying the decisions, you know, and decisions you need to make. What I find useful is just list all the decisions you think you need to make, and look at the ones that you really have to make today, and look at the ones that you can just kind of have your backlog of decisions.

[00:24:01] Henry Suryawirawan: And this is coming back to the thing that we normally called ADR, Architectural Decision Records. So is this something that you also advocate for every team to have ADR for any architectural decisions?

[00:24:13] Murat Erder: ADR is a technique. So I advocate that you should have your architectural decisions. If ADR has worked for you, the benefit of it is also in your Git repository and it’s quite tightly coupled to your code is benefit, and it’s just part of the way the team works. So there’s lots of strong things in ADR. We do use ADRs in my current role as well. So they’re quite powerful on that perspective. But it doesn’t have to be ADRs. We just create a simple Kanban boards for architectural decisions. So it depends on what works for you culturally. The main thing is you have to have a log of your architectural decisions that can be ADRs. That can be a Kanban board. That can be something else, whatever works for you.

[00:24:53] Understand Technical Debt

[00:24:53] Henry Suryawirawan: Thanks for clarifying that. The next essential activity is around actually knowing your technical debt. I think this term is like abused sometimes. What is actually technical debt in your point of view?

[00:25:03] Murat Erder: So you have to have a view of what good looks like. And that is like, there’s a coherent view of the architecture of the product. That doesn’t mean it’s in one person’s mind, just the architect. It’s probably the team has that view. Technical debt is decisions you make to get something into production that could have an impact on that coherence, and potentially impact sustainability of the product. I’ll give an example. We built a data integration layers. The ideal thing is that data integration layer is that there’s a standard interface and all consumers get that data from that standard interface. But certain consumers are not able to do that. So we actually, on our side, just to make sure that we can support those consumers and get the system live, we do that transformation on our side. So we have, I guess you might call legacy set of feeds to serve a set of consumers. That is our responsibility. So we’re maintaining two sets of interfaces. The interfaces, what you would call, those strategic future set of interfaces, and you have more legacy set of interfaces and technical debt.

That’s an example that we consciously make a decision that we will support these things because otherwise, our product, which is our data product in this case, at a very high level, would not work because we’d have unhappy consumers. Because our product cannot satisfy the consumers. But again, we’re making our architecture more complex. So that’s an example of technical debt. You can have other examples of technical debt that impact your scalability. You might go with a certain architecture that is not as scalable, but is quicker to develop and deploy.

[00:26:34] Henry Suryawirawan: So, also the other aspect of technical debt, it’s not just knowing, but it is beyond that. When actually you should reevaluate your debt and actually pay back those debts. So do you have a view on that?

[00:26:45] Murat Erder: Yes. The main view is that, and this goes back to the way of work in Agile, right? You have to have with your stakeholders that are at the end of the day, each software product meets a certain set of requirements from a certain set of stakeholders. Let’s call them the business. You have to prioritize what you’re going to work on with the business. The discipline you have to get into is to have the technical debt log, and then when you speak to your business, just say, okay. Here are the features we’re going to implement, but also here’s the technical debt we want to reduce in this next sprint.

The main thing is, that goes back to another aspect of architecture is to be able to communicate effectively. I think there’s the architecture elevator pitch, which I know Gregor was, I think on one of your former podcast. It’s about the ability to communicate at the level that’s appropriate. So the big thing of dealing with technical debt is to be able to articulate why something is suboptimal and why should it be fixed? The discipline is to have your log, and make sure you have the discipline in each sprint or each cycle that you deal with your business stakeholders to say, okay, what I’m going to knock off this technical debt log, and you have to be able to articulate why it’s important versus something else.

[00:27:50] Henry Suryawirawan: I see this is one of the main challenge sometimes about technical debt, because it’s easily said that, “Okay, we have technical debt, we have technical debt, and we should work on that. We should pay it back. We should fix it.” But actually quantifying that and communicating it to the business with the stakeholders that this is actually a priority, right? What is your advice actually, how to quantify this? How can you explain that actually this matters? Otherwise, this will break. This will not scale. This will create problems. How do you actually quantify this debt?

[00:28:17] Murat Erder: With great difficulty, right? That is, I think a challenge we do face in the industry. You know, we use this term and as you said, you’re kind of right. I have had lots of business stakeholders love that term. Because for them, it’s, oh, let’s just do this technical debt. We’ll just write it down, and they just move on. So, it’s really to be able to articulate the impact. And back to the quality attributes we discussed, technical debt is doing with the quality attributes and also the complexity of the product. And again, all of that doesn’t have to be paid back. You can maintain your legacy interfaces for a long time, if you want to. It’s just additional effort.

Going back to the legacy interfaces example I gave, if you have 10 interfaces, and one is on the API. You want the nine are on the legacy, and those legacy systems keep on changing and you have to maintain that on your side. You know, you can articulate that actually, more than 50% of my team’s effort is wasted on maintaining the legacy. So we have to do something to refactor that so that I can say, it’s only 20% of my team’s time is invested on maintaining the legacy. So it’s to be able to articulate it in terms that the business understands. And I think quality attributes they kind of understand, but the main thing they understand is percentage of effort the team has to spend on maintaining sub-optimal architecture.

[00:29:33] Data as an Architectural Concern

[00:29:33] Henry Suryawirawan: So, let’s get the other two. Because due to the time, maybe we will not be able to cover everything. So there is another important thing that you cover in the book. Just now when you mentioned about architecture decisions, right? There is one specific thing that tends to be so hard if you do not choose. Which is actually coming back to your data architecture. So thing like you choose the wrong database, like you choose something that doesn’t scale. And in order to migrate data, change the database, I think it’s very tricky. So in the first place, you mentioned that data is actually an architectural concern. Why is it such that data is actually a major architectural concern in your view?

[00:30:09] Murat Erder: Yeah. So data is an interesting topic, right? We use it so much. It’s almost lost its meaning. What do we mean by data and how do you visualize it? There’re multiple comments I think we put from Tim Berners-Lee in that data chapter. Like the data we have will always outlive the systems. You can think of it as if you’re running just a mail-order catalog system. Originally, you would have written it in such a way that you would ship catalogs out. People would send things to you or phone you up and you have a system that supports that. While then, over time, it might have gone to CDs being distributed. And over time, you know, we’re now in obviously the internet age with the more Amazon type of shopping cart model, etc. But it doesn’t matter. The data is still the same data. You still order a component. If you were a company like that, you would have had four or five generations of different systems supporting the same, but the data is still the same. So data is core to the way we operate.

A big change that’s happened in the last 10 years has been the introduction of new database technologies. 20-25 years ago, if you built a system, it was almost no question that you would have a relational database. The big debate was whether I use Oracle or Sybase, or SQL, whatever it was, but that was there. Now, mainly due to the big tech companies that deal with the big scale, the problems of the internet, there’s lots of new, not-only SQL solutions there. They address lots of good technology aspects, but your data is much more integrated into your architecture than it was in the past. If you use a key value store, or documents store, or a graph store, it does impact the way you think about your application and how it operates. And it really deals with the quality attributes we set up front. Those are the things that are much more difficult to change. You can split your components, you could do lots of things. You can refactor your code a lot. But changing the underlying data structures you deal with and some decisions that you made on that are fundamental. They’re much more invasive than they used to be in the past, because in the past there was this nice separation of relational database and object-oriented analysis code. And that is no longer true.

[00:32:10] Data Technology Trends

[00:32:10] Henry Suryawirawan: I like the way you explain this, because data is always outliving the system. We can always rewrite the system, do monolith, microservice, whatever frameworks, technologies, and all that. But the data is always there. It’s really hard to actually change it. And you mentioned one of the specific key trend in data technology space, SQL versus NoSQL, right? There are different types of database. In fact, database technologies probably is one of the hottest area these days where people just create new and new database, and try to solve different problems depending on the systems that you’re building. What are some of the other key trends in data technology space that you can see that change all the complexity of this data as architectural concern?

[00:32:48] Murat Erder: First of all, there’s one trend that I see probably more in financial services and pharmaceuticals, and it’s not a technology trend, but it is focused on data governance. Defining your data correctly and making sure that the business stakeholders take ownership of that data. There is another big trend that’s coming up is around the data warehouse, which then evolved into the data lake, and other things. Now there’s, which I think is a very powerful trend, and we don’t refer to in our book actually, but it is as recent as the topic of data mesh. If you think about it, there’re two types of systems. There’s the more operational systems and the more data analytics systems. I think the operational systems with microservices and the things that we’ve talked about, we know how to build them better.

But when you look at a corporation, then there’s this big schism, and then your data analytics world used to be the big data warehouses, some of the proprietary like Teradata and things like that, then we’ve got into the Hadoop world that everyone built data lakes, and you might apply new technologies, but that’s still is a monolith, right? That is like a monolith. So I think a big trend is the data mesh concept that came out from the ThoughtWorks team. And that is very powerful because it’s about applying the same concepts you would to build an operational system to your data architecture. It applies some of our core principles because you start thinking this data as a product. What’s my data as a product? That’s very important. And you are speaking of architect for a change in your leverage power of small, because you’re trying to break up that way the data is managed.

Obviously, we don’t have enough time to go into the detail. I would recommend people to go and read the articles on that thing. It also talks a lot about principles six, which is the organization of the team. Because you have the challenge with these big traditional data analytics infrastructures is the fact that you have various specialists and those specialists are centralized, and they have to manage everything and they don’t have enough domain knowledge because the technology is so specialized. Those specialists focus on that. So when you tried to implement this data mesh concept, which is in some way of creating components around data products, you also want to make sure that those teams have the technical knowledge, or have access to that technical knowledge to be able to support their data products. So I think the data mesh concept is very powerful. It’s not the technology trends. It’s more of an approach to dealing with these big data architectures.

[00:35:03] Henry Suryawirawan: Yeah, so another data approach that I can see a lot, especially with big scalable internet companies, is what we call eventual consistency. Things where you don’t need to have atomicity. Coming back to the concept of RDBMS world, where everything is ACID. But these days, a lot of people and systems are built based on eventual consistency. Some are even built based on event driven architecture. Maybe there’s a thought from you about these two approaches of building data intensive application?

[00:35:31] Murat Erder: Yeah. First of all, I recommend the book that is great on data architecture. It is by Martin Kleppmann and it’s “Designing Data-Intensive Applications”. This is a very, very useful book, especially if you’re interested in the technical detail of how to dealing with data. Eventual consistency is interesting, and it’s been around for quite a while. And actually, if you think about it, our real life is eventually consistent. I could be talking to you right now, and there would be a traffic accident in front of my house at this moment. But I don’t know about it until I go and walk out into the street or a neighbor comes and tells me about that event happening. Again, it’s the type of application you’re building. If you’re building a very traditional transactional application, I think the ACID principles and what the relational databases offer to you are so strong that it’s kind of unnecessary to ignore that.

Sometimes what we do is we go after fads, right? So we try to know that NoSQL around. We try to make sure that any system we built has a NoSQL component. But it doesn’t have to be, right? You have to think what’s right. There’s a very strong thing about having a ACID compliant database that you trust that it can manage the state of your application for you. That’s a very powerful tool. Because the moment you step away from that, you have to do much more engineering yourselves. You have to say that it’s worth the effort due to the nature of the software product I’m building and the nature of the quality attributes that I wanted, that I go to a world that’s more distributed, eventually consistent.

Another trend is this whole event sourcing. Think of basically saying, I don’t need a database to maintain my state. I just capture all the events that happen. And if I can play back the events, I get the state of my application. These are very powerful techniques. If you have very high writes and higher reads applications that have lots of scalability concerns, but they do require additional engineering, which is something you have to think, is that worth the effort?

[00:37:20] Importance of Data Model

[00:37:20] Henry Suryawirawan: And there’s actually one other important aspect of data, which I think you mentioned also in that chapter, is that for us to have a well-defined and communicated data model, as a prerequisite for successful software development. I think people refer back to Domain-Driven Design, understanding ubiquitous language. So why do you think having a well-defined data model is actually very important?

[00:37:41] Murat Erder: I think it’s key, actually. I mean, there’s two aspects. One is the common language. And again, there’s the ubiquitous language in Domain-Driven Design. We have to have a common language when you talk. Not only the team that’s developing the software, but the common language with the business stakeholders. It’s quite important because it’s very difficult to understand what problem you’re solving if you don’t speak a common language. It sounds very high level and superficial, but it’s quite important.

In my day-to-day work, for example, I’m working a lot in the new area of skills and how skills are important for an individual as well as for the organization. It gets very nuanced. What do you mean by a skill versus a competency versus behaviors and values? Those types of things. People, when they talk about it generically, their own mental map of those terms. If you don’t define that, and it doesn’t have to be a detailed schema or DDL, it’s really just have a common language to talk to that you end up in circular conversations, you might implement things that don’t meet the business requirements, etc. Having a common language for a software product, actually across software products that you support for a business, is quite important.

And the second aspect of that is done for each data entity, knowing which application is mastering that, and which one is consuming it. Again, a very simple thing, but if you don’t think about it upfront, you can end up with very confusing data integration points between multiple applications and what we call data daisy chaining, right? You get data from another application, you slightly modify it, you send it to a third application. They modify it, and they sent it to a fourth. And before you know it, no one really thought about who owns which attributes or which identifiers, and you end up in a mess because you start looking at them, you have actually an inconsistent representation of your business. Think data first is very important.

[00:39:27] Henry Suryawirawan: Yep. So I think that’s why it’s very important. And a lot of people are talking about Domain-Driven Design these days is because of this importance of ubiquitous language, having the same data understanding. Thanks again for mentioning about the masters and the consumer of the data, who actually owns the data and has the right to actually governs the data. Coming back to the data governance side that you mentioned earlier.

[00:39:46] 3 Tech Lead Wisdom

[00:39:46] Henry Suryawirawan: So thanks again Murat, for all the fascinating talks. But unfortunately, due to time, I have one last question for you before I let you go. Normally I ask every guest that I have to share the three technical leadership wisdom. So this is just for all the listeners here to learn from you. So what will be your three technical leadership wisdom?

[00:40:04] Murat Erder: Okay. The first one is the team. Really work with the team. We are social creatures and software is about working with your core team and your wider stakeholder. So, team first. That’s very important.

The second technical wisdom is that technology fads come and go, but the core problems we solve are still the same. So really think about what I said, what’s the core of architecture? Assigning responsibilities to certain components, and why you’re doing that? That’s what really it is. Really think about that. What am I asking that component to do? That’s very important, and that is the crux of all the problems we’re trying to solve.

The third one is don’t underestimate the importance of collaboration and communication. It’s very important that we have to constantly with us, other technical people and our business stakeholders have to continuously collaborate, communicate, and agree on a common way forward.

[00:40:55] Henry Suryawirawan: Thanks for the wisdom. I like the second one. Technology fads come and go, but the business problems are actually all the same. So for people who are interested in looking more about the book, where can they find it online? Or maybe can they follow you on social media?

[00:41:10] Murat Erder: Yeah, we have three coauthors. So it’s myself, Pierre Pureur and Eoin Woods. You can follow all of us on social media. We do post on LinkedIn from time to time. We also have our continuous architecture website. So just go to continuousarchitecture.com. So that’s us that has all the articles that we’ve published and background on the principles. And then finally, the book is there as well.

[00:41:32] Henry Suryawirawan: Thanks again Murat for your time today. So looking forward for the next time we meet probably sometime in the future.

[00:41:37] Murat Erder: Okay. Thanks a lot, Henry.

– End –