#176 - Acing the System Design Interview - Zhiyong Tan

[00:01:06] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I have Zhiyong here with me, the author of “Acing System Design Interview”. Hearing this title, I’m sure you know that we are going to be drilled by Zhiyong about how we can do system design better. If you are a more, maybe like software engineer, right, you would have maybe experienced this in your interviews before. And it’s always stressful, at least for me, because sometimes you don’t know what to expect. And in the end, you also don’t know whether you are doing great or not, right? So hopefully Zhiyong can give us some pointers today. So welcome to the show, Zhiyong.

Zhiyong Tan: Thank you for having me, Henry.

[00:01:43] Career Journey

Henry Suryawirawan: Right. Zhiyong, I always love to start in the beginning by asking my guests to share a little bit about maybe your highlights or turning points that we can learn from you.

Zhiyong Tan: Well, I suppose if this was advice for the general software engineering community, yeah, I would say one point is that you should work with your manager. Well, when you enter a new team, one of the first conversations you should have with your manager is to develop a career roadmap. This might include the projects that you would be working on as well as training opportunities, whether it’s a formal training from the company, training that the company sponsors for you or training with other software engineers within your team or within the company in general, the more of an informal type of training. So yeah, following through with discussing the progress of the roadmap that, and any changes that you will make along the way, it can be pretty dynamic. It should be one of the main discussion points in your one-on-ones.

Personally for me, I’m from a non-traditional CS background. So I suppose one of the turning points was during the time when I was trying to break into the field. And yeah, I was interviewing while learning simultaneously. And I would say it takes a lot of resilience. And this is something that you should, yeah, I guess all of us should continue to develop and continue to remind ourselves that we will and we should continue to face and seek out challenges. Because it is these challenges that help us to grow to develop technical skills, to develop project management skills, develop communication skills, develop conflict resolution skills, and understand about system designs as we seek out like new challenges, like new types of projects and unfamiliar domains.

And as I mentioned at the beginning, when I talked about breaking into the field and coming from a non-traditional CS background and having to interview and learn at the same time, yeah, this taught me the value of resilience and it also gave me the attitude that I kept to this day. To always keep learning and always to keep seeking out new challenges. And I think, I hope that the spirit of this is embodied in the book that I’ve written. I hope that it would help, it would make things a little easier to give people a sense of direction which I myself lacked. So yeah, I would say this is one of the… I’m not quite sure if I describe it as a turning point exactly, but this is some advice that I would give to everyone.

Wow, it’s very inspiring about your story, right? Starting from a non-CS background, actually, trying to, you know, go to the industry. Software engineering can be really daunting at this time, where so many things to learn. And not only that, now you have successfully written a book as well about a topic that some software engineers probably are like afraid to do when they go to interviews, right?

[00:05:03] System Design Interview

Henry Suryawirawan: So system design, it’s one of the rounds in any major big companies, tech companies, right, that people have to go through. But in the first place, maybe, because the definition kind of like not so well-defined, what is actually system design interview? And I know that some companies define it differently or maybe do the kind of test differently. But maybe in your view, to level set our understanding, right? What is system design interview?

Zhiyong Tan: Well, it is an interview. It is not about coding, obviously, but it is where you are asked to design a scalable and efficient software system to solve real world problems. These interviews, they are commonly conducted for engineer roles, especially for the more senior roles where you are expected to have a strong understanding of architecture and design principles. You will typically be given a problem statement or scenario and you’re asked to design a high level architecture to address requirements. And then you would have to deep dive into topics like scalability, reliability, performance, the non-functional characteristics that is mentioned in, I believe, chapter three of the book.

And the goal really of a system design interview, yeah, is about assessing your problem solving skills. How you analyze a complex problem, how you identify requirements, and how you design an appropriate solution. It might be about evaluating your design skills, your architecture and design principles: scalability, modularity, flexibility, maintainability. To a lesser extent, it’s about testing your technical knowledge in terms of such as in distributed systems, databases, networking, caching, load balancing, cloud infrastructure. Yes, these knowledge is good to have, and in fact, more important as you become more senior. But really, it is about…, at the end, it’s about your communication skills, how effectively you communicate your ideas, how you explain your design decision, how you justify your choices to the interviewer, how you evaluate possibilities and discuss trade-offs. This really is about your ability to think through various possibilities and discuss them effectively. It is really what the interviewer is looking for here.

Henry Suryawirawan: Just by listening to what you said, right? It seems there are plenty of things that we all need to take care of, preparing before we even go to the interview, right? So I think a few key takeaways from me is that, yeah, the first one is like we will be asked to solve real world problems. Sometimes, it’s a bit imaginary from time to time, right? Depending on the interviewers. But it’s mostly solving real world problems, right? Second thing is about, you know, understanding the problem, right? The problem solving skills. Some system design probably asks you like a generic, high level, abstract kind of question, then you have to deep dive from there, right?

[00:07:36] Trade-offs

Henry Suryawirawan: And I think the third thing which I want to go much deeper is actually about the trade-offs. So I think you mentioned in your book that a system design interview is all about trade-offs. Tell us how can we actually think about this trade-off? Why is it important in system design?

Zhiyong Tan: Well, regarding trade-offs. There are many, sometimes, endless possibilities in designing a particular system. Well, firstly, yeah, you have to understand what the customer is asking for both the functional requirements and the non-functional requirements. And then from there, why the discussion about trade-offs comes in. Well, you don’t want to overengineer, you don’t want to build something that the customer does not actually require. And another consideration is that everything you build, there are always a cost involved. If you were to make a system more real time, for example, it would be more complex. It would be more expensive. And you are assuming that this real-time performance is something which is actually needed by the customer.

So for instance, if you could make a system eventually consistent or you can make it asynchronous that the system does not have to immediately return an updated value in response to user’s input, there are certain technologies and certain techniques that you can use that would considerably reduce the cost, the complexity. It might improve availability, but as I mentioned, when we talk about trade-offs, really it is about talking about various approaches that can all solve, the, well, address the requirements, to a larger or lesser extent. And then talking about what you would prioritize and what you would deliberately pay less attention to, so as to ensure that the costs and complexity is kept and maintainability is kept under control, for instance, while you are satisfying the other non-functional requirements like scalability, availability, security, privacy, for instance, to the desired extent.

Henry Suryawirawan: So I think very important in the beginning, you mentioned understanding about the requirements first, right? Either the functional and the non-functional requirements. So without that, probably, it’s very difficult to actually come up with the trade-offs, right? So solving the first problem is about the requirements. And then we have to be able to explain.

[00:09:51] Managing the Time

Henry Suryawirawan: So this is probably the tough part, because, first, the time is kind of like limited. Some people maybe only have like 40 minutes, right? Or some have one hour, full one hour. But most likely, it’s going to be limited, right? Especially the nature of open-ended discussions. And the third thing is actually about communicating it. So maybe some tips here, how to manage time and how can actually you communicate the trade-offs or your decision and priority better in the interview?

Zhiyong Tan: Okay. With regards to time management the interviewer would deliberately give you a vague question. And, yeah, you have to agree on the scope of the discussion. And that is how it is kept contained within an hour. Yeah, it is impossible to discuss a complex system in one hour. You got to narrow down the scope. And the interviewer is looking for how you discuss what the requirements are, the functional requirements and the non-functional requirements. Yeah, so you would spend a few minutes on that discussion. And then, after that, only then you will start discussing various possible approaches and their trade-offs. And then, most likely what’s going to happen is that you would do a deep dive into one or two of those choices. And this would involve sketching high level diagrams onto the whiteboard.

Well, when you sketch down, like, the high level diagrams into the whiteboard. Yeah, it is definitely possible to keep things really high level and sketch it out really quickly and gloss over many of the details. And then the rest of the interview would be about discussing this design at the desired level of abstraction that the interviewer is interested in. Or which you, the interviewer, might also ask for your opinion into which are the critical pieces that you believe are the most interesting pieces of the design that you can do a deep dive into. And then from there you would be drawing more detailed diagrams of specific components of your system.

So this is how you would have such a discussion within 40 minutes or one hour near the end of the interview. The interviewer might start asking for additional requirements, or he might push a certain non-functional requirements to the extreme. For instance, pushing how do you serve a 1 billion users? Or how do you handle millions of requests per second? Or you might start discussing about how your system would tolerate, how your system is resilient and how it would… what are the graceful degradation characteristics? If there’s certain dependencies fail or if certain parts of your system fail, how that would be handled?

So yeah, that is definitely endless topics to discuss. But within the 40 minutes to one hour, if you are able to clarify the scope, clarify the requirements, and design and architecture, well, discussed several approaches, the design and architecture that can satisfy the discussed requirements and weigh the trade-offs of your approach, that would be generally sufficient for the purposes of the interview.

Henry Suryawirawan: Right, I like in the beginning that you mentioned that the interviewer intentionally gives you a vague question, right? I think we all have experienced this, like they give just one, two sentences. Design me, I don’t know, like Uber. Okay, or design me like Google Maps or something like that. I think the key here is to ask clarifying questions, right? Don’t just assume you know what they are asking. Always ask in the beginning, probably, right? Have a back and forth. What to prioritize? Because the scope can be super big and you have to agree on a level set of understanding, right? Like what is the scope, what is the priority for the interview session?

[00:13:27] Handling What You Don’t Know

Henry Suryawirawan: And I think just by listening to the question, right? Some of us will feel anxious, actually. Like okay, I don’t know the answer fully but I also don’t want to give no answer. How do you actually manage this kind of stress and anxiety? And actually also sometimes you don’t know the answer altogether. Maybe some tips from you, how we can manage all this.

Zhiyong Tan: Well, the interviewer is generally not looking for domain knowledge unless the job description specifically states that this domain knowledge is required. And you are unlikely to be building the system that is asked during the system design interview. So that’s not the purpose of the system design interview. The purpose as we went over is to assess your communication skills, assess your understanding of system design, in general. And so yeah, you are not expected to know everything.

So when you come across like areas of a particular system where you do not possess a certain domain knowledge, right? You can start by assuming, there is a black box that accepts inputs and returns outputs. And you can state your assumptions about this box, both the functional and non-functional characteristics of this box. And you can say that this is your starting point in how you approach this question. And you can investigate further about like how these particular systems are designed and implemented. Yes. So that’s typically one approach.

Another approach is you’ll be asked to possibilities of how would you design and implement a particular domain which you are unfamiliar with. You can start building from first principles. So for instance, like if you are asked to design a pathfinding for Google Maps, or if I asked to design a rate limiter, yeah, you can start by designing an algorithm for a single user. And then consider how to scale it to a large number of users. What data you would need and how you would handle the data. You could then discuss trade-offs of your approach. The interviewer might sometimes actually help with the domain knowledge and then the question, then the discussion then becomes like how you take that knowledge, how you critically evaluate the approaches that the interviewer presented to you, the pros and cons, and how will you build a system around it. And it’s also about asking the right questions.

[00:15:40] Managing Anxiety

Henry Suryawirawan: Then how about managing anxiety, like do you have some tips to like, I don’t know, do the preparation, maybe a good rest, or maybe, I don’t know. Like do you have some tips that we all can learn how to manage performance anxiety in the system design interview?

Zhiyong Tan: Well, experience definitely helps. One of my suggestions would be to practice with your friends, with fellow software engineers. Or sometimes, perhaps even with people, non-technical people within your company. But maybe even with your friends or family who are not in the tech field at all. But however, who are interested to help you succeed. This is how, in fact, a pretty valuable way for you to train your communication skills. And this is something that is sometimes also assessed, your ability to explain complex technical concepts to non-technical people, which is actually a pretty relevant skill on the job, particularly as you become more senior. So, yeah, I suppose, yeah, that would be one good way you can handle performance anxiety. Keep getting as much experience as you can.

Well, another way would be to increase your knowledge. So the system design interview book and other resources like it are a good starting point. They can present you with some templates and some mental models that you can use as a starting point for most system design questions. And if you were to follow the industry of topics that you find yourself to be more interested in, for example, in a database design or development. Or AI or LLMs, which is the hot topic right now, if you were more interested in infrastructure side of things in DevOps, for instance.

Then I think continuing to learn and continuing to gain knowledge and continuing to discuss your learnings with other software engineers, would give you the confidence that you need when you are entering a discussion, where you are discussing a topic where you do not have like all of the understanding or all of the details. So yeah, after a while, these discussions would become natural, more natural to you. And yeah, you would become accustomed to handling unfamiliarity, which is in fact something that you should get used to progress in your career. And yeah, so these are some tips that I would give to handle a performance anxiety.

Henry Suryawirawan: Yeah, familiarity and, you know, getting used to being asked these kind of questions, I think is very important. Especially for those senior who have already day job, right? So you work on a particular system, most of the time, you know from the front to the back, right? But system design is all about preparation, I think. And resources like your book, probably, and maybe a few other online resources, you can also use them as a guide like templates and mental model. I like how you say that. So I think it’s very good to have preparation. And also, yeah, don’t forget to actually have a good rest, you know. Like be humble because you may not know everything. So I think just accept that you probably won’t be able to answer everything.

[00:18:32] System Design Interview Principles

Henry Suryawirawan: So in your book, you also cover a few principles, which I find really, really useful and insightful that can give us like a guide how should we approach system design. Maybe if you can mention, I know that we have covered some of them. Maybe is there any other principle that you think we all should bear in mind whenever we go to system interview?

Zhiyong Tan: Well, I’m not sure if I would like to approach the topic this way and insist on a certain dogma because that is the antithesis of what a system design interviews are or what system design is, in general. I suppose that if I were to push like the most general principle, it is all about trade-offs. Just always remember that. It is not about perfection. It is about trade-offs. And yeah, your ability to understand this, the pros and cons of any approach that you choose to be able to discuss many possible approaches, to be creative, and to be conscious of the current and possible future requirements. Yeah, I think this is about what it is really and being able to communicate them clearly and concisely.

So okay, if I were to summarize what I just said, it is really about trade-offs and it is really about clear communication as you mentioned. If I were to add in one more thing, you will not know everything. And you will not expect others to know everything. Everyone is continuously learning in our industry, in the tech industry. Yeah. New technologies emerge. And they do so in a rather unpredictable pace. These trends and yeah, startups just suddenly pop up all around them, whether it is about machine learning and the internet of things or metaverse, blockchain, and now about AI and LLMs. These are topics we all need to continuously keep learning them. And understand how they are applied to systems and the trade-offs and we’d be able to communicate clearly about our thoughts.

Henry Suryawirawan: Right. So thanks for the very great tips, right. So it’s all about trade-offs, understand the current and future requirements, communicate it clearly. I think communication for some of us can be very daunting, right? How to explain your technical thought process into something that other people can also understand. What I find sometimes useful as well, right, you probably can understand what the problem that the company or the team is solving and try to relate back from there. Because I think in some interviews, I experienced at least, they’re trying to solve a problem or they have solved a particular problem and they just want to validate their approach with you, right? So maybe for some of you who can maybe do some research first, what the company is doing, what the team is doing, and try to come up with some various different combinations of answers, potential answers that you can prepare. And also don’t forget the ultimate tips. Try to ask around from people who have done the interviews at that particular company. I think it’s always useful.

[00:21:22] Non-Functional Requirements

Henry Suryawirawan: So maybe let’s move a little bit to the non-functional requirements. I think probably this is one of the difficult topics in the system design interview, simply because, again, there are many trade-offs. Maybe tell us a little bit more about non-functional requirements. Which non-functional requirement we should be prepared a lot more? And which one that you think is something that interviewers always ask?

Zhiyong Tan: I believe the topic of scalability pretty much almost always comes up about how you would cost efficiently and how you would design your system to easily and cost efficiently adjust its hardware utilization to serve the request load. That is one. The first few that I mentioned in the book, scalability, availability, performance, latency, throughput, yeah, so these are some of the most common topics that you would discuss. But then you would also talk about the trade-offs that you can make, the complexity, maintainability, debuggability and testability of your system, and how you would trade off like these certain non-functional characteristics. For instance, can you make your system less available in order to reduce its costs? Would they satisfy the customer’s requirements? The customer often, both in the interview and in real life, they would not be fully aware of like these non-functional requirements and what they can possibly sacrifice in order to optimize for certain other characteristics.

I didn’t mention consistency. So well, do you need a strong consistency or is eventual consistency good enough? When a particular user writes data to the system, must these updates be made available immediately at a high cost and high complexity, or can the system be designed to propagate these updates to be non-blocking, to be asynchronous, to propagate these updates after some time, so as to save on hardware costs. Do you actually need to serve every request or can you apply a sampling or can you apply approximation techniques? Yeah, this can also save on costs.

And well, to a lesser extent, you may need to talk about the security and privacy characteristics. Most of the time, in my experience, this is not a strong focus point. But what is important is that you express your awareness of these requirements and you are able to discuss them for some time. Yeah, well, discuss them to the desired level that the interview desires, should these topics come up.

So yeah, I’ve spoken a lot about non-functional requirements. And in general, it’s hard to pin down exactly what would be considered important and what is considered unimportant. I believe within the book and within many similar resources, they have already condensed the non-functional requirements to this list. And this is the most…, well, this is the list of important requirements or characteristics that one must discuss about a system. And should there be other non-functional requirements? Yeah, this can be something that might be discussed like near the end of the interview as a bonus topic or probably not at all.

Henry Suryawirawan: Yeah. So I think, yeah, just what you mentioned, right? The first few that we probably need to focus a lot more on is like scalability, availability, right? How to make your system maybe 24/7 up. And…

Zhiyong Tan: Or how not to, yeah.

Henry Suryawirawan: Oh yeah. And how not to, if it’s allowed by the requirements. And the performance, right, your speed, latency, right? Also like throughput, how many requests per second. And sometimes it’s always helpful to be able to know how you calculate number, like request per second, right? Because the interviewer might just give you like, okay, millions of users a day. But that doesn’t mean like a million requests per second, right? So you always have to come up with some explanation. How do you derive that request per second? So I think this is all very common topics. Security, privacy, also in my opinion, unless the company really deals a lot, I don’t know, like with secure data, you know, like PII and things like that, probably they won’t ask too much. But yeah, do prepare. Know about OAuth and things like that.

[00:25:11] Data Consistency

Henry Suryawirawan: So I think in your book, you cover quite a number of various topics, right? From the simplest to the most difficult one. And one of the probably difficult one to also answer is about the consistency. So CAP theorem probably is commonly asked or ACID principle, right? So maybe in your view, how do we tackle consistency much better in the system interview?

Zhiyong Tan: Well, the interview is not, firstly, it’s not about testing whether you know the dictionary definitions of CAP theorem or ACID. That would not be any, well, in my opinion, not too much value derived from being able to recite these definitions. But you are not going to think that one candidate is better than another, because the first candidate was able to recite like this knowledge from rote while the other candidate cannot. As I say, it is really about the awareness of these principles, in general, and how they are applied to system design and what trade-offs then are applicable with regards to these principles.

Yeah, so about, as you mentioned, ACID. So ACID is applicable to relational databases. And you would talk about ACID or the requirements for ACID if this is something that this is in the list of non-functional requirements that you need strong consistency, in particular. And then you would talk about the trade-offs of having ACID or having a relational databases. For instance, in most cases, in most implementations, all of the data has to fit into a single host, a single write replica. So the writes are typically pretty difficult to scale, but the reads are fairly easy to scale. But if you were to follow the approach where the reads are easy to scale, where you start implementing read replicas, then the system is no longer, it gives up the strong consistency property. It becomes eventually consistent as the updates take some seconds to propagate.

So then just being able to talk about these trade-offs, this is something that is part of, well, system design discussions, in general. And if a system did not have ACID properties, well, we start talking about NoSQL. And the idea of NoSQL is, the general idea is that you are making some trade-offs against ACID to optimize for other properties of the system. In particular, being able to design a distributed database more easily. If you were to try designing a distributed database with relational databases, yeah, that has been done multiple times. However, it typically comes with pretty heavy trade-offs.

In summary, yes, certainly, it is essential to understand the overall idea and purpose of various common paradigms like CAP theorem or ACID. But the main importance is not in knowing the exact definitions, but in knowing how they are applied in system design.

Henry Suryawirawan: I think that’s a very important thing, right? So be aware about all these, right? You don’t say like, I don’t know about all those. But be aware, try to be able to understand at the high level. But yeah, hopefully the interviewer won’t dig you deeper into definitions, right? I know that in your book, you also explain that this book is not particularly just for the people who want to do system interview, but also for the interviewer to get some tips how we can do system design better, right? Because there are variations of how interviewer does the system design. Some focus a lot more on theory, some focus a lot more on practicality, some focus on imaginary problem. So hopefully by reading this book, interviewer will also get a good idea how we can do system design better.

[00:28:41] Database Scaling Problem

Henry Suryawirawan: So you just mentioned a few things about database. In my experience, database is always challenging, right? It’s probably the first bottleneck that you will see whenever you build a more like scalable performance system. So from your point of view, how should we start thinking about solving database related topics in the system design. Because most likely, you will start with a single database and then you, because of some reasons for the requirements, right, you handle like lots of requests. Is there a thought process how we can tackle this database scaling problem?

Zhiyong Tan: Well, the system design interview for the purposes of our discussion, well it’s not about designing databases themselves, to clarify. That would be a totally different discussion, but it is about selecting the correct database for your non-functional requirements. Again, it is about trade-offs. Database choice is all about trade-offs. And yeah, you would typically start with, if you were to say you’re going to start with SQL, it is typically because of the simplicity. So you would consider making your system less complex. It would be for a system that is starting out with a few users and, uh, you’re trying to bring it to market quickly, prove the product market fit. And as you scale up, as the request, the number of users or the request load increases, then you are going to have to start making some trade-offs against the ACID properties of the relational DB to optimize for certain use cases, the most common use cases of your system.

And then this is where you start talking about the various types of databases, of NoSQL databases. And yeah, there are various properties and trade-offs. Are you going to go for in-memory, a key value database like Redis? Are you going to go for a database with columnar storage like Cassandra? Or are you going to go for a database that is unstructured, that’s key value but unstructured, like MongoDB? And this would typically depend on your specific non-functional requirements. Is your system more read heavy or is it more write heavy? For instance, that you will select the appropriate database to suit that requirement. What is the latency requirements, the P99 requirements? Must every user request be responded to immediately? Then you will start looking into in-memory databases such as Redis. Or can it be eventually consistent? Can it be asynchronous?

That is actually one of the main questions that you will ask, you will try to ask. You will try to discuss, like which parts of your system need to be real time and which parts do not need to be real time, they can eventually become eventually consistent. And then you can start introducing event streaming approaches like Kafka and then Kafka feeding to a HDFS. And this would considerably reduce the cost and complexity as compared to trying to build a real time, high performance, low latency system. And for approaches that when if you’re not able to do this, then you will have no choice, but to increase the cost and complexity overall.

And there’s one more thing I wanted to mention. Lastly, we talked about sampling and approximation earlier, and this is certainly something you should try to bring up, even if it might seem obvious to you that the… sometimes, yeah, you should not make assumptions, as we say. Don’t make assumptions in the requirements. Always clarify. If you can do sampling and approximations, then this dramatically reduces the scalability issues. And you would most likely, in the purpose of the interview, you would probably discuss the possible scenarios where you can apply, what happens if you can apply sampling and approximation? How will your design change versus what happens if you cannot go with this approach, yeah, what will be your design choices then?

So yeah, in the end, it all comes down to discussing various possible requirements, especially non-functional requirements and how your database choice would be different and how the overall system architecture would be different with regards like to different sets of non-functional requirements and the different trade-offs that you are trying to make.

Henry Suryawirawan: Right. So I think, uh, you cover a lot of things that are very, very important and useful, I think, about database, right? Because I think not many people have the luxury of experiencing, you know, working on systems with complicated database scaling kind of a challenge. So I think in your book, you have covered a number of topics, which I’m sure some of us, if you read the book, you will get some good ideas how you can tackle these questions much better in your next interview. So always check out database problem, because I think that’s one of the key criteria on how you can actually make your system more scalable, more available, and more performant.

[00:33:09] Distributed Transactions

Henry Suryawirawan: So I think another important thing these days is about designing your services, right? And also handling distributed transactions, so to speak. So many people these days talk about monolith versus microservice, right? When you have a lot of services, how do you ensure transactions are kept as atomic as possible, probably, if that’s possible. So how do you tackle this kind of distributed transaction questions?

Zhiyong Tan: Well, it helps to know a few of the general approaches that you would take regarding distributed transactions. And yeah, that is covered in the book. So if you were to know these few high level approaches, you can bring them up. But before you do that, again, you should clarify whether you need distributed transactions at all. You should explain what the purpose of a distributed transaction is and why you believe it will be needed for your system. And then you will explain throughout the possible approaches to implementing it within your system design. In my experience, you did not necessarily have to go into the specifics of anything more complex than showing in an event streaming such as, Kafka. And then, yeah, feeding that to a database, well, a distributed database for further data processing, downstream processing.

That is typically as far as you would go. But it would definitely be valuable to learn about orchestration versus choreography approach of distributed transactions. It shows the depth of your understanding and it shows the awareness that you can bring up these couple of approaches and talk about their pros and cons and how that would solve the issue of like having to ensure consistency between multiple services in your overall service-oriented architecture.

And Henry, as you mentioned, the trade-offs between monoliths and microservices. Well, in today’s world, this comes with a very big asterisk or caveat, almost all systems are microservice architecture and it is unlikely you’ll come across monoliths. But you should be aware of the trade-offs. And in fact, there are certain systems where a monolith will make sense. For example, Amazon’s, their video streaming, where they refactored from microservice architecture to monolith and that resulted in a considerable performance improvement and improvements in costs. But the decrease in costs, however, yeah, it came with all of the trade-offs of a monolith, such as possibly having to scale the entire system when you develop any part. And the speed of deployment and the speed of the overall, the flexibility of the system to changing requirements. These are some of the trade-offs of monoliths.

Yeah, so really, it all comes down in the end to, again, improving your general understanding of these high level approaches. Be able to discuss their trade-offs. And it would definitely help to have a passion in some of these topics and to follow the industry developments, to read about these and discuss about these blog posts. Having a passion for your job as a software engineer, it is certainly something that would bring you very far.

Henry Suryawirawan: Yeah, I find that these topics probably is researched and well published online, right? So don’t forget, always read up about all these distributed transactions, event-driven architecture, synchronous versus asynchronous, right? And also maybe like orchestration versus choreography, like what you mentioned, right? So some of these topics are well covered, so please do prepare as well.

[00:36:31] Functional Requirements & API Design

Henry Suryawirawan: And I think one particular thing that I probably missed in the beginning is asking about the functional requirements aspect, right? How should you explain your understanding about functional requirements? Because like you mentioned, maybe domain knowledge is not necessarily interviewer asked from us, right, to understand, for example, how do you create like a banking application. But how do you actually satisfy functional requirements understanding? Is it about API design? Is it about design patterns? So tell us more about it.

Zhiyong Tan: Typically, it is about understanding the specific set of use cases that you will be covering in designing this system. So an API design is one way that you can approach that. You can scribble down an API, a REST API spec really quickly, but it does not have to be perfect. Yeah, you definitely don’t need to get every detail. You’re not there to test like how well and how fast you can define a REST API. But rather you’re there for the interviewer to understand that you can take a vague requirement. You can discuss the set of specific use cases that the customer actually wants. And those are the functional requirements that you are going for.

And yeah, based on the functional requirements, then you would discuss the non-functional requirements that a customer actually needs. And from then on, you would design a system that would fulfill all of these requirements. And the discussion then it’s typically about your design choices and possible alternatives. And yeah, the trade-offs that come with them. So that is how you would approach it.

Henry Suryawirawan: Yeah, so be able to design the API, at least, right? So what are the requests that the system will ask, right? And what are the outputs, I think it’s very important? If you do understand high level business workflow, I think that’s also important in my view. It’s always a bonus, right, if you can always understand about the high level business workflow. And think about all the different cases that a particular system would need to handle. Things like, for example, I don’t know, reconciliation, reporting, and you know, handling peak surge of the requests and things like that. I think it’s always very, very important.

[00:38:38] Failing System Design Interview Henry Suryawirawan: So, Zhiyong, we talk a lot about all these topics. So sometimes we do fail system design interview. Maybe from your tips, how should we handle, you know, being suck in the system design interview? And how can we actually make it better the next time? So any tips on handling failure here?

Zhiyong Tan: Hmm. Well, firstly, we talked about resilience at the beginning. And definitely, you would definitely go through many failures. You would definitely fail more interviews than you would pass. But perhaps, there are a few of us for which this rule is not true. But yeah, I’m making an assumption on the behalf of the overwhelming majority of us. And I can think of like some generic advice.

Certainly, as I said, be resilient. Don’t be discouraged. And yeah, you have endless chances. You can keep trying. There are many companies that you can interview for. And you have like the rest of your career to keep getting better at what you do. And really it helps to be passionate as a software engineer.

It would help if your thoughts about it isn’t really just in terms of how well you can do in it as a career. Well, these are definitely considerations that one should have. But if you are driven solely by this, then you’re going to have a much harder time in dealing with failure than otherwise if you do have a passion for this industry, for this line of work and in coding, in system design, in building systems in general, and in collaborating with others to realize like large projects, to bring large projects to reality. So remember that this is your passion, this is your purpose.

And along the way, there will definitely be setbacks, not just in system design interviews, but in the job as well. There are many things that go right. There are many things that go wrong. But the important thing is that you try your best to collect feedback, to learn from your successes and your failures. And yeah, to really keep challenging yourself and really keep learning. And if you were to have this attitude that you bring to interviews, that you bring to your work, you would definitely go pretty far and your career path would be paved with accomplishments.

So in summary, how you would deal with failures in system design interviews? You would analyze like what you believe you could have done better. Any places, any particular parts of the interview where there were specific topics where you lack certain knowledge or where you failed to discuss possible approaches and their trade-offs. Generally, you would practice, like you would take this question, you would practice it, you would examine it thoroughly. You would note down what you did right, what you believe you did right, what you believe you did wrong. And then you can discuss it with your peers. And read up about the relevant topics. Maintain a good attitude about it. And yeah, this is how you’ll keep pushing forward and how you will succeed.

Henry Suryawirawan: Thanks for highlighting about resilience one more time, right? So I think system design interview, majority of people would experience some kind of failure. I failed in the past as well. I think of many times, right? I think system design interview, sometimes you prepare something, but then suddenly you are expected to design something else. So exposure I think is really important, right? Not necessarily exposure as in like hands-on experience, but also exposure by reading, understanding, you know, real world challenges. You know, every time a tech company publish a certain blog about solving a particular problem, I think, do read up. Check that out. Sometimes it can give you some kind of insights how you would think certain problems differently.

[00:42:02] 3 Tech Lead Wisdom

Henry Suryawirawan: So, Zhiyong, it’s been a great mentoring about system design. So thank you so much for sharing your insights. For people, do check out Zhiyong’s book as well. There are so many real world problems, questions that are being covered in different chapters in the book. So it’s highly dense, like so many, many topics in one book. So I think you will get a lot of things, like insights, like how you can prepare system design interview better.

So I have one last question that I would like to ask you. This is something that I call three technical leadership wisdom. So if you can think of it just like advice you want to give to the listeners, maybe can you share what is your version of three technical leadership wisdom?

Zhiyong Tan: Well, if the question is about technical leadership, in particular, firstly, I would say the three things. Continuous learning, firstly, continuous learning. Secondly, your communication and your collaboration. And lastly, leading by example.

So yeah, regarding, continuous learning. Yeah, what you know today might become outdated tomorrow. So you have to stay curious. You have to stay committed to learning. You should encourage your team to engage in ongoing education through courses, workshops, conferences. Keep staying updated with industry news and trends. So your team will remain agile and adaptable to new challenges, to new opportunities. And you should be kind, you should be humble.

And this is a culture that should be pushed into the team that, as we said, not everybody knows everything. People have different strengths and weaknesses as well at different particular domains that one is passionate in. And in order to succeed as a team, everyone, every team member should complement each other. And then you can bring your combined knowledge, your combined expertise and experience to really realize like something which no team member can accomplish alone. So yeah, continuous learning, I’ll say is the top thing that we have to keep emphasizing.

Another thing, communication and collaboration. Embrace open dialogue within your team, ensure everyone feels heard, everyone feels valued. Ideas should be freely exchanged. Constructive feedback must be welcome. Conflict should be addressed promptly and respectfully. And this is particularly important in the tech industry, right? We are innovators. We are building. We are creators. And in order to realize like something which the world has never seen before, we got to harness the collective intelligence of our team to drive this innovation.

And lastly, yeah, leading by example. As a tech leader, your actions are louder than your words. So some examples of what you might want to happen within your team is technical excellence or to have strong ethics and integrity. You should show your team what it means to be passionate, what it means to be dedicated, what it means to be hardworking. You should be transparent in your decision making process and you should take ownership of a success and failures. And by setting this positive example, this is how you will inspire your team to really bring out their excellence and to cultivate a culture of accountability and trust.

So, yeah, these are the three technical leadership principles, which I like to follow and which I believe are essential for a high performing team and company.

Henry Suryawirawan: Thanks for sharing such great tips, right? I like the continuous learning part, right? So. technology these days, very rapid change, right? So AI, LLM, like you mentioned, is the hot topics. It can be overwhelming. But yeah, don’t forget to always learn, maybe keep up by reading. But also don’t forget, you don’t have to know everything. Again, this is a very important thing in technology. Probably you won’t be able to know everything, so just be humble, be kind, and always support each other.

So, Zhiyong, if people would like to learn more, maybe is there a place where they can find your resources or you yourself online?

Zhiyong Tan: Well, I am currently working on a couple of personal projects. And I hope that everything that I have done, well, when it comes to my professional work or within like this book, the book Acing the System Design Interview, all the personal projects that I’ve released to the general public, it would really convey my passion, the spirit of my passion and my love for what I do that really comes out in that.

Yes, but if you would like to keep track of what I’m currently doing lately, you can take a look at an app called Tingxie. It is an app that I’ve written for my daughter and it’s for learning or preparing for Chinese as a second language exams. I am trying to apply a new technology which we did not have when we were kids, with in particular, yeah, new developments in smartphones, in touch screens, in text-to-speech. These innovative technologies that were not available before, into the experience of, learning Chinese as a second language. And, uh, yeah, I hope that this work also brings out, like one can sense the passion and love for what I do in the product.

Another project that I’m working on is something called jointgoals.com. Yeah, I am helping out technically with a team to build a financial advisor for scaling family finance advisory to the general public, beyond the 0.01%. Yes, so tech is after all about improving accessibility. It is about improving quality of life, and it is about making what was previously luxuries or what’s previously exclusive, like available and more accessible to everyone. And, yeah, I hope that this is something which I will continue to believe in, I’ll continue to be passionate about.

And, uh, lastly, if you wish to discuss the contents of my book, you can always go on the Manning forum. And yeah, I generally respond usually within a few days. But yeah, if a topic particularly piques my interest, then yeah, we can have a pretty lively debate there.

Henry Suryawirawan: Okay. Thanks for those useful apps, right? I’ll make sure to put it in the show notes so that people can also benefit from it. So thanks again for this conversation, Zhiyong. So I hope the listeners here can learn a lot of tips about how to do system design interview better. Thanks again.

Zhiyong Tan: Okay. Thank you Henry.

– End –