#238 - AI is Smart Until It's Dumb: Why LLM Will Fail When You Least Expect It - Emmanuel Maggiori

[00:01:27] Introduction

Henry Suryawirawan: Hello, everyone. Welcome back to another new episode of the Tech Lead Journal podcast. Today, I have with me someone very excited, to, you know, share some learnings or some insights that he has been dealing with regarding AI. I know maybe this is like on everyone’s mind these days, right? Emmanuel Maggiori actually wrote two books related to AI. The first one is called Smart Until It’s Dumb. And the second book is titled The AI Pocket Book. I find this a rare opportunity for us to learn more about AI. What actually these days, you know, general- generative AI, LLM, all these tools that we have been hearing a lot and using a lot in our day-to-day work, try to understand the basics and maybe few interesting things that Emmanuel can share today for us to learn from him. So Emmanuel, thank you so much for this opportunity. Welcome to the show.

Emmanuel Maggiori: Thank you for having me.

[00:02:32] Career Turning Points

Henry Suryawirawan: Right. Emmanuel. Uh, I always love to invite my guest first to maybe share a little bit more about you by sharing any career turning points that you think we can learn from you.

Emmanuel Maggiori: Career turning points? Yeah, I think there were two important ones. The first one was when I realized that a lot of the technology industry was driven by hype. You know, a lot of projects were just, I don’t wanna say fake, but a little bit fake. Sometimes, not on purpose. But you find yourself working on products or on building products that don’t make any sense and everybody knows it, right? There’s a difference between an experiment that is worth doing, right? Okay, we don’t know how the client will react to this, so we’re still going to build it as a sort of MVP. There’s a difference between, you know, that, and just building something that doesn’t make any sense. Everybody knows it, but you’re still building it because a VC told you that it was a good idea, or because an upper manager from a company just gave you a lot of money to do this and then you’re just doing it.

And unfortunately, I found myself involved in projects like that a little too much. And at one point, I thought it seems like most of it, especially when you work on cutting edge stuff, it’s like that. And I think it was a turning point because I was very naive before, right? I thought companies are efficient. You know, why would a company ever carry out a project this long when it’s obvious that it will never work. But you know, they do. Things like that can last for a very long time and these projects, um, they get swept under the rug very easily. So I think that was an important career turning point.

And I think another one, which, you know, well, that first career turning point led me to the book Smart Until It’s Dumb, what you mentioned, right? It was, I need to tell the world that AI is not that great. That there are lots of problems with it. That lots of projects fail. And why? So that was why I wrote that book, right?

And then another turning point was, you know, I thought I’m gonna work on more foundational stuff, less hyped up stuff. Just do some more traditional software engineering. But then I realized that a lot of people weren’t working much. They weren’t doing anything, right, including me. I found it hard to find work to do. I was hired by a company and I thought it was going to be super exciting and they kind of put me in this staging area to wait until I was assigned to a project. And there were lots of us in there being paid to do nothing, right? And that, you know, I thought something’s really strange with this field. It seems like there’s a lot of money flowing into the field, lots of projects. They get launched without really knowing what, you know, they’re gonna be or what we’re gonna do. And companies like to hoard staff sometimes. And I thought, again, this is companies not being very efficient and I didn’t think it would last.

And then everybody got fired, right? Twitter fired 80% of its employees and everybody. You know, Meta, all major companies, made a lot of people redundant. And I thought, you know. I mean, some people work very hard, but a lot of people, they, they’re just, they just don’t know what to do. When I thought that wasn’t sustainable. So that was another learning for me.

Henry Suryawirawan: Wow! Very interesting sharing that you just shared just now, right? So I find, I mean, when you said that I also had experience, uh, of those kind of, you know, projects, career, right, where, you know, seems things are going to the wrong direction, but we still march on anyway. We think that something positive will come out of it. And the second one, you know, especially for consulting or professional service kind of work, sometimes, yeah, we were put on the bench or on the beach, some people also call it. And maybe in some big organizations, there are roles which have maybe little things to do and, you know, spend their time doing, you know, something else which is not necessarily productive. So thanks for sharing that personal journey that you have so far.

[00:06:41] Writing “Smart Until It’s Dumb” and “The AI Pocket Book”

Henry Suryawirawan: And I know the books that you’ve written is something about AI, right? So probably you can share a little bit of background why you wrote those two books. Like what kind of things that you actually want to convey by writing these two books.

Emmanuel Maggiori: So I think my first book came out of this frustration of everybody saying that AI was the future. We should be AI first companies. And this was, look, I wrote the book before the ChatGPT thing happened. But ChatGPT was released when I was finishing the book. So I added stuff about it, but it was kind of about that time. But what I’m trying to tell you is that I’ve been noticing this before, even that wave of AI hype that we witnessed. And I wanted to tell people how AI actually worked, you know, how machine learning works and why it has some issues, let’s say. Um, there were lots of, you know, it was a very broad book. I even spoke about consciousness or the mind, you know, because you have all these people even telling you that we’re just building artificial neurons. But in reality, we don’t really even know how neurons work very well. So how can we actually be building them?

So I thought it was, it’s good PR, but it wasn’t quite true. And I wanted to also convey, if you want my experience in the business of AI, because I’ve seen so many projects fail that identified a way in which they fail. Almost the same way every time, repeatedly the same mistakes. And I thought, I need to tell this to the world, right? So it was a kind of an antidote to AI hype. That was my, the goal of my book.

[00:08:14] The History of AI Booms & Winters

Henry Suryawirawan: Right. So, yeah, I remember back then, I dunno, like 5, 10 years, uh, ago, right? So there are other fields of AI compared to what we know now, which is like generative AI, LLM, you know, all the craze about these kind of AI. But in the past, there are so many AI technologies as well. And in fact, in your book you mentioned about two AI booms that actually happened, but actually it ended up going nowhere in the end. So maybe tell us a little bit of this history so that people know about the context and they can relate to what they’re experiencing now.

Emmanuel Maggiori: Yeah, so there was an AI boom in the sixties, which is kind of crazy because computers were new, right? And they weren’t very powerful. But people were saying all these things, uh, everybody will lose their jobs in 20 years. You know, this is something that was said in the sixties. And there was a whole boom around AI. The idea was to build those sort of artificial worlds, right? That you could have a world where you simulated some physical things in a computer, and people thought that would have an impact, but it didn’t, right, in the business world. So after that, there was what’s known as an AI winter, which means a time where there was a very little money in AI and very, uh, little enthusiasm about AI.

Then in the eighties, there was another boom, right? And not millions invested. And this was about expert systems, which is when you try to encode all the knowledge of a person as rules inside a computer. It didn’t work well. It was impractical to do. And there was another collapse, if you want, of the AI field. Another bust of the bubble. It also became almost shameful to say that you worked in the AI field at that point, right? People said, stop saying AI. They started saying, I work in cognitive something, ‘cause they didn’t really want to say AI ‘cause it was stigmatized.

Then we had after that period of, let’s say, kind of calmness, there was a much more moderate, if you want, boom of AI, uh, that started to happen in the 2000s. In 2010, it intensified, right? And that’s the one that kind of I lived through. There was a difference here though, which is that AI in the form of machine learning. So this was the machine learning boom, right? The computer learns how to do stuff from data. And the difference is that this time AI is successfully used in commercial applications for the first time. You have your Netflix recommendations and there’s AI there because the computer’s learning from your behavior and also what other users that are similar to you like, and then you get personalized recommendations. AI will sort search results when you’re shopping for a holiday, when you’re searching for things on Google. So AI has been with us for a while, and it’s been used in a commercially successful way, right? And that’s when I thought people were exaggerating about the power of AI already. ’ Cause they were saying we need to use it for absolutely everything, right?

And then ChatGPT happened. That’s kind of the a new wave, yeah. Yeah, I thought there was gonna be a sort of AI winter again before ChatGPT. And there were signs of it. There were some industries that were struggling, especially the self-driving car industry, for example, that was doing really, really badly and still is. But then with ChatGPT, everyone forgot about that, you know? And it was like the most impressive boom that we could ever imagine, yeah.

[00:11:34] Why Generative AI Hype is Different Than the Past AI Waves

Henry Suryawirawan: Yeah, so I think, I feel that, you know, all these hype about ML, I experienced it maybe back then 20- , I dunno, 2010, 2015, those kind of era where you start having like, I dunno, vision, OCR, those kind of things. Recommendation system also starts to appear. And then maybe a few cloud computing also productize their AI services, right? So I think that’s when some of the craze about AI, you know, people are trying to use that. Especially also a lot of startups, you know, cool disruptors, so to speak, right? They wanna also implement AI use cases.

And I think, yeah, this time is really so much different. You know, the ChatGPT, the generative AI era. So what do you think makes it such a craze now? Because, uh, you know, you have studied the history before. Um, but this time it seems so real and so different and people are even talking about AGI and all that stuff.

Emmanuel Maggiori: I think there are a number of things. So the first one is that this new generation of AI is used by the general public directly. Just go into ChatGPT and use it. The previous generation wasn’t like that. You know, as I said, there’s AI recommending you movies on Netflix, but you don’t get to see any of that. You don’t use it yourself. But now with ChatGPT, you are using it directly. So people have access to AI in a much more straightforward way. But also, it’s conversational AI. And that’s a big difference too. And there’s so much fascination around having a conversation with a machine. And it’s also one of the most difficult tasks historically. The whole, the Turing test is that. It’s like can you build an AI that is so powerful that you can’t tell whether it’s a human or not? And this test is a conversation where you chat with an AI, right? So conversation is fascinating and it has so many applications, and I think that’s also a crucial difference with previous generation of AI.

[00:13:26] AI is Smart Until It’s Dumb

Henry Suryawirawan: Yeah. And I find the barrier for you to use it is so much easier now, right? So you can just go to a website or even embed it in your phone these days, right? And even you can speak to it the assistant way, uh, just like what you mentioned, right? I think, yeah, so many people just try, and they believe it’s pretty smart. Your book is titled Smart But It’s Dumb. So why do you call it? So I guess that’s the first thing about AI, right? Because people think it’s smart, but actually, maybe there’s another side to it that we don’t know.

Emmanuel Maggiori: I think that that was pretty much the thesis of my book, which is that AI works really nicely and it’s impressive, until it’s not. Because you discover… I called it at that time an epic mistake. Now it’s called a hallucination, right? People, that’s the name that people have adopted for it. But I’ve been warning people about this for a very long time, which is that everything’s nice with AI until it makes a mistake that a human would never make. Ever. And this reveals that it wasn’t actually smart, right? And those things again are, they’re very difficult to quantify and to even, you know, define. But intuitively, all of a sudden you’re like, okay, this is really, really dumb.

And I’ve been observing this for a long time and I thought this was, for example, the thing that would kill the self-driving car industry. And it is the thing that did kill the, it’s almost dead. Like, uh, Cruise cars, you know. The brand Cruise that builds self-driving cars, they have more people in a control room than self-driving cars. And they operate them remotely. They don’t really operate, they react to problems, right? And they happen so often they need more people than cars, right? The other company, Waymo, they also have a control room and they intervene every three to five miles of driving manually, right? And I knew this was gonna happen. I’ve been saying it for years.

And the problem is that machine learning learns by repetition, learns by example, right? So you need to show examples of situations and then the computer kind of interpolates. If it seemed to similar examples and there’s something in between, it can make deductions, but it’s not very good at processing situations that are far from anything the computer has had access to the data. And when you’re driving, there’s so many things that can happen that are completely out of the ordinary.

And for some reason though, we don’t really understand, humans are very good at that. We seem to have these internalized models of the world that we use to process things, to analyze things that, you know, you see something totally different you’ve never seen but you know how to react to it. And with machine learning, it’s notoriously bad at that, right? And I said, this is going to be bad for the self-driving car industry. And that’s what’s been making the adoption of AI so hard now. Even now, generative AI in the enterprise world, right? Or even to code, to code in a serious way. People say, oh, it does 80% of the job, but then I have to go and debug the thing that AI created, and it’s causing me more trouble than if I’d done it myself, you know. Not always, but there’s a, and it’s again, these hallucinations. And that’s why I thought smart until it’s dumb is good way to summarize this, right? Kind of describes the whole situation in a very short sentence.

[00:16:45] How LLM and Generative AI Actually Work

Henry Suryawirawan: Yeah. Yeah, I think almost everyone, including software engineers, right? I mean, we are, I mean I perceive us as someone who understands how a computer works and maybe we know about logical thinking and things like that. Even us are still kind of like impressed sometimes when you use coding assistants. You know, how come it can generate a code that seems to work? I mean, it works. There’s a vibe coding as well, right, these days. Uh, it seems to work. But yeah, there are times where we think it would work, but sometimes it just fail epically maybe, uh, borrowing the word that you mentioned. And many, many catastrophic mistakes actually happening as well, like deleting production database and things like that.

So definitely there’s something that we, like I personally, even though I use AI a lot, I don’t understand the fundamentals. Like I don’t understand how all these LLM, generative AI actually works. Maybe it’s also a good time for us to understand a little bit, uh, about LLM, right?

Emmanuel Maggiori: Yeah, yeah. Yeah, sure, so we need to… um, a language model, very simple computer program that tries to predict the next word based on the previous word. Very, very simple. And if you wanted to produce an entire sentence, you just make it eats its own output. So you generate one word or a token, should be a piece of a word from a vocabulary, it generates a piece of text, you add it to the prompt, then you run it through the program again, right? That’s all it does. Anything else is a sort of illusion built on top. So for example, when AI seems to search the web, there’s actually another program, a wrapper around the LLM, which intercepts an instruction that says please go search the web. And then you go and you search the web with another computer program. And then you gather some results and you put them in your prompt and then you continue. The user doesn’t see this, right? But under the hood it’s just a system to predict the next word.

And it was trained on two types of data. The first one is a huge amount of data collected from the internet. So if you want, this learns how to guess the best next word, according to what internet have said. There are lots of problems have been solved by people. The solutions are online. And you can see that this can take you pretty far but not completely far. The system will not gonna learn how to do math, for example, ‘cause people are not saying online two plus four equals six with all possible numbers for the AI to actually get it. So what AI is gonna do is going to create a plausible output. Something that looks like a solution but not an actual solution ‘cause there’s no data to support it. Same thing with the names of citations of legal cases. There’s this lawyer uses ChatGPT to try to find legal precedence for a case. And it kind of invents names that sound good, but they’re fake. And I’ve seen that a lot of times.

So the second stage in training these AI models was to use manual data created by humans. So what they did was they asked people to generate lots of responses, and they ranked them manually. They said, this is a good response, this is a bad response. They did a lot of, you know, trying to make it more appropriate, not say inappropriate things this way with people, you know. Paid very little money. Some people say they hired people in Kenya for $2 an hour to do a lot of this manual work. And then all this data was fed back into the machine. This is called reinforcement learning with human feedback to improve the models, make them more aligned, made them better at solving well-known problems, right? And that’s how they’re trying to fix the hallucinations and all these problems. But there’s always like a long tail of problems that never disappears. But yeah, that’s the essence of how…

And another thing I want to say, why, why now? Why, why does… because the way this computer program works is different from what people were doing before when they created language models which already existed. In the past, they were using something called an LSTM which is a very naive type of technology, I think. I was always a bit skeptical of it. ‘Cause what it does is processes one word at a time, and it tries to kind of summarize the entire context up to now in one little vector of numbers that tries to express what’s been said so far, which is a huge compression of all this information into a tiny little thing. And then you use that to predict the next word. But you’ve pretty much, you’ve forgotten a lot of stuff. It’s very, very hard.

So what they invented was this so-called transformer architecture that takes the whole context and tries to represent the meaning of every token in the context simultaneously. They try to disambiguate the meaning of tokens by kind of cross-referencing them, and then it predicts the next word from a really, really rich representation of the context. And this is what changed from previous technologies. And this is something we’ve seen a lot in AI, which is that they, somebody invents a new approach and that’s when performance is unleashed. It happened with convolutional networks for image analysis, right? And now transformer architecture for text generation.

Yeah, so it kind of is in a nutshell how AI works. Very simple principles, very powerful, but also not all powerful.

Henry Suryawirawan: Right. And when people refer to generative AI, is it referring to the same thing as LLM or is it something slightly different?

Emmanuel Maggiori: Well, that’s complicated, first of all, because in academic circles, generative AI does not mean what people usually mean by generative AI. So forgetting about the technical statistical definition, just generative AI means AI to generate things. So it could be LLMs when we’re talking about language, but it could be also images, right? And that’s not an LLM necessarily, although you may be typing in something or, you know, when you’re transferring this style from an image to another, you could call the generative AI, but there’s no language. When you’re typing in, there is a language model somewhere in between, but you’re generating an image. But what I would say is it’s mostly about generating new content based on some cues or some prompts that we give to the machine.

[00:22:53] What Makes LLMs Smart

Henry Suryawirawan: Thank you for the clarification, right? So every time I read about the basics of LLM, right, the primitive understanding is that, yeah, it tries to predict something based on a given input, right? And you just multiply it with so many inputs and maybe you train, you kind of like do reinforcement learning and all that. But somehow it just works magically, you know. If you use some kind of assistant or maybe ChatGPT, they will say, thinking, right? Thinking and then gives you some explanation of what it tries to do, which if you read about it seems very natural, like a human thinking and trying to deduce stuff. And even we use it for coding. Like I’m still amazed kind of like how come it can produce code in a certain framework, in a certain language, that mostly, mostly, uh, works syntactically correct and with a good output and things like that. Like maybe explain us a little bit. Like I’m still kind of like don’t understand in between how come it can be so powerful and looks really smart.

Emmanuel Maggiori: I think that internally, those computer programs, right, the LLMs, they have states. They’re called hidden states. We can think of a hidden state as a sort of abstraction that we don’t necessarily understand what it is. We let the computer learn its own abstractions, right? So for example, the LLM may learn that to produce or to make good predictions about code. ‘Cause that’s what it’s trying to do. It needs to have some sort of internal state that tries to produce syntactically correct code. As in it will maybe count the number of opening parentheses, you know, so that it matches the kind of the closing parentheses. So that’s what made it, makes it really powerful.

There are many layers stacked on top of each other that have hidden states that try to, when the model is trained, these hidden states configure themselves if you want, in the best possible way to solve the problem. And because there are so many layers of them, we can’t really understand where they are, if we look because it’s gigantic, right? But I would say, because some people, they become a little too critical of AI and they said it’s stochastic parrot. It just imitates or copy paste stuff from the data, the training data. Not really, right? What it does is it uses the training data to try to configure those internal hidden states.

Initially, uh, I remember asking ChatGPT, does an Anaconda fit in a shopping mall, right? And it told me that it didn’t because it’s such a large animal, right? What this shows you that there was no hidden state that properly represented the size of things or what things are big or what big means, right? It was still producing good text in the sense that it sounded correct, because its hidden states had properly represented the grammar of English, right? But not necessarily the sizes of things.

But then this became quite viral and as they improved the algorithms, they probably generated manual data. I have to believe that they did that probably somewhere where people asking questions about the sizes of things and manually ranking the answers and that was fed back to the LLM. And at some point, the LLM must have created an internal hidden state that tries to represent the things that are big in general, right? And when things fit into other things, stuff like that. We can’t really know what it is ‘cause it could use shortcuts.

There was a famous case of this image recognition algorithm that or model that failed to detect a cow on the beach. And the reason was that a cow was not on grass. So the algorithm had actually learned that a cow is this thing that looks like a cow and the grass underneath, because all of the images and the training data had grass, right? So it didn’t really learn what a cow was. So they can, and that this also causes hallucinations, right? The AI may not really learn what we want from it. However it does have hidden states that try to represent things in an, with a higher level of abstraction, I would say.

And because they’re stacked on top of each other, you can combine them, right? So there’s a first layer where it tries to represent some low level things, right? Is this a noun? Is this a verb, right? At the next level, it could represent, is this a noun that represents a big object or a small object, right? And after lots and lots of layers, you get a very, very good representation of the meaning of things. And I think, yeah, the key to the power of this is all this hidden stuff that goes inside the machine.

[00:27:25] Foundational Model

Henry Suryawirawan: So I understand like sometimes machine learning, you can’t really explain what is happening because it goes through, I don’t know, like thousands or maybe millions iterations, and that’s why probably these hidden states are kind of like embedded inside the model, so to speak, right? And maybe now it’s also a good time to understand about what is called a model. Like people are calling foundational models, you know, OpenAI, Gemini, Claude, whatever. I dunno how many foundational models are there. So maybe if you can explain what is foundational model?

Emmanuel Maggiori: Okay, let’s start with model. A model is essentially a computer program that tries to predict something. So it’s the essence of this AI. There are different reasons for the use of the word model, we could say it’s trying to model the world, but I’m a bit more practical with that. I don’t like to use the word program, ‘cause there are two programs in AI.

One is the program that trains the other program. You have a program, right, which is this model which an AI engineer will configure, but they leave holes in it. Things that are undefined that are called parameters, which are numbers, right? So you create a sort of template of the computer program, and then you need to run your training, which is how to scan all this data to try to fill in those blanks in the template. And that’s another program. And if we use the word program, we can, we get confused. So which one is which, right? So typically you’ll use the word model to represent the final results of your training, which is this program with all the blanks filled in, right? And then you use the word training algorithm or something like that to explain the alg-, the program that actually does the training, right? So have a computer program building another computer program. The last one is the model.

And I think with foundational models, people mean very general purpose LLMs that can do lots of things, right? And they can be used as building blocks for other AI. So let’s say you want to build some, an app that is based on AI. You may connect using an API to a foundational model, right? Or you may take the foundational model that is open source and fine tune it to your own data so it works better on your specific problem, right? The foundational model would be very, very generic. And then you have your specific model to do what you want.

Henry Suryawirawan: Right. So I think I’m always very fascinated by people who are producing this foundational model. I can’t really imagine how they train it because it’s such a general purpose. You can use it for language, you can use it for translation, math, whatever people wanna try, right? I think it seems to be quite smart and powerful.

[00:30:01] RAG and Agentic AI

Henry Suryawirawan: So these days, there are new techniques that people are trying out, you know, things like RAG and also agentic AI. So maybe if you can elaborate these also shortly and maybe what other things that are new that probably we are not familiar with.

Emmanuel Maggiori: Yeah. RAG means retrieval augmented generation. It sounds very fancy, but it’s not. It’s essentially a following. You have, let’s say in your company, you have a database with a lot of your own private data, right? And you want to search, using, use AI to search or analyze that data. So what you do when the user queries something, they search for something, or they ask in a prompt for something. You run a separate algorithm to go and find relevant documents that could help. And then you insert the text of those documents in the prompt. The user doesn’t see this, but you say, okay, these are the documents. So you need a large context window. You need, uh, a model that allows a lot of or very long prompts. You insert all this data and then you add instructions. You say, you know, answer the original question referring to this data. What this lets you do is not change the foundational model, not fine tune it to your data, right? So it’s a way to specialize an AI to your own company data or any specific data without changing the model. So yeah, that’s RAG.

And the challenge is, okay, how do we find that data? How do we find the relevant stuff? But a good application of that is what we see with Google AI results, which tend to be terrible by the way. But anyway, we get to see AI using websites and analyzing websites. And the way it probably works under the hood is that first Google finds relevant websites using its old algorithms. And then it takes the text in those websites, puts it inside some prompt somewhere, it says answer this query using also this information, right? And then the LLM will generate answers which are then intercepted and shown to the user. So that’s RAG.

And then agentic AI or AI agents, I feel like there’s a new word every day, um, coming up. And now agentic seems to be the new word. An AI agent means that there’s an AI model that you use in a pretty autonomous way and you execute some actions that come out of the model probably in an automated way. So for example, you create a little program that uses an LLM to generate some instructions. And then you use that instruction. I can imagine, I don’t know, imagine an agent that scans your emails of a corporate person who travels a lot for business and you tell the LLM whenever you detect that this person wants to travel somewhere, output this. ’ Cause the agent can’t really do it. It’s like output an instruction that says search for flight for these dates in this specific format. Then you connect that to an API that goes and searches for the flight, for example, right? And then you get the results and give them to the LLM and you say, you know, send an email to this person with a proposal of this could be your flight, right? So essentially automate workflows and maybe even connecting multiple AI agents. But essentially every agent is nothing but an LLM to which you send a fancy prompt.

Henry Suryawirawan: Right. So I think, um, when reading your book as well, right? Explaining all these, uh, fundamentals thing about the AI, LLM, things that we understand now as like a cool thing, right? So definitely you can understand some holes that could probably, uh, poke, you poke, right? And you can start seeing, okay, why sometimes it hallucinates or it makes mistakes, right? Simply because of the way it is, I dunno, the fundamentals of how it works, right? And maybe we can find clever ways of doing things like the RAG and maybe the wrapper that you mentioned, right? Don’t forget. I think the wrapper is also very, very powerful, I see. Because, um, you know, you can tweak the workflow, so to speak, within the way AI replies the response to you, right? I think the wrapper is kind of like the intelligence as well that people build on top of the foundational models.

[00:34:09] Tips on How to Adopt AI Within Companies

Henry Suryawirawan: So I think thanks for highlighting all these fundamentals. So when people talk about AI these days, people think about, I need to use AI. Businesses also scrambling so hard to implement, adopt AI. First maybe adopt like ChatGPT and all that within their companies. And also building something smart, you know, using AI. And I know in your book, you mentioned, you know, there are some failure modes that could happen. So maybe tell us how can business or organizations, you know, start thinking about adopting AI or using AI within their companies?

Emmanuel Maggiori: Yeah. Well, there are a few things. The first one is that I don’t think the best way to do anything is to have the hammer and search for nails. You know, saying, what can we do with AI? I’ve met companies that created a whole team to promote the use of AI in the organization, and it didn’t go well because they were very biased. They were, for the survival of the team, they had to say, this can be done with AI, right? And they didn’t acknowledge the limitations of AI. In my experience, most AI or most failed AI projects start with that saying, oh, what can we do with AI? So I wouldn’t do it that way. Uh, it’s good however, to be aware of what AI can do, what the limitations are, because it’s just another tool in the toolbox, right? So when you find a problem, something that could be done in a more efficient way in an organization, you can see if AI will fix it.

I think the best way to use AI is to acknowledge these hallucinations from the get go or just the limitations in general of AI. Because if you embed that into the, your product, you can build a successful product. If you forget about that, you build a product that people won’t like. Because at one point, it would be like, you told me this could do this job, but then it hallucinated, you know, when I least expected it. So one way to recognize the mistakes, for example, is to have AI retrieve or point to specific paragraphs of, let’s say, documents and then show you the para-. So it’s so much different when AI interprets stuff than when you use it to actually point you to something, for example, right? Because it could help people search for things more efficiently, right? And if AI gets it wrong, then you still very quickly you can see, you can say, okay, AI said this paragraph proves something, but it actually doesn’t, so I’ll keep searching. So I think we need to embed the idea that AI is not perfect in the way we use it, and that’s how you succeed with AI.

And another thing I think is, um, as a developer or in general, in general we want to use AI when you can describe your problem very succinctly. Because if you’re doing something so difficult, so custom that it takes you longer to describe it in the prompt than doing the job, you don’t gain anything. So it needs to be something that’s very generic, very easy to describe, and then it needs to be easy to validate, you know, the output. A typical example is you, you know, you’re coding and, uh, you already know how to do this or you’ve done it before but you forgot how to do it, right? So you go and ask ChatGPT, how can I, you know, read a file from S3 using this Boto3 library and then a very easy to describe. And then ChatGPT outputs few lines of code which you’ve used before but you’d forgotten how. And you can also very quickly validate if it’s working, so it’s fine.

Some problems are very difficult to describe, right? And, uh, and in that case, I think AI is not necessarily going to save you time. And also sometimes the output is very difficult to validate. Something you don’t know at all or a very complicated algorithm. And then you’re like, you need to just read someone else’s code, someone else being AI. And, uh, you know, you’ve read other people’s code and sometimes it’s difficult to follow. So you need to make sure, you know, to use AI in those easy to describe, easy to validate scenarios, I think.

[00:37:56] How to Reduce & Avoid AI Hallucination Problem

Henry Suryawirawan: Yeah, I think that’s, uh, pretty good tips, right? Because I still think that companies try so hard to implement AI for smart products, right? So especially these days, like, I mean, almost every product embeds something called AI features inside it. It could be, you know, summarizing something and things like that, right? But the most classical use case is actually chatbot, right? The conversational thing. Because it has been around even before LLM, right? So I think a lot of chatbots. And now, uh, with LLM, I’m sure people are still trying hard to create chatbots, customer support thing. And we have seen many, many cases where, you know, the agent responds something wrongly or incorrectly, sometimes even give, you know, wrong result to the customer such that it mislead them or the company, right. So I think, definitely there’s this risk of hallucination. How can companies deal with this hallucination? So is there something within their solution or within their implementation that they can do in order to reduce this hallucination from happening?

Emmanuel Maggiori: I mean, you can reduce hallucinations. There are techniques. You can improve your prompts. You can fine tune your model if it’s not working well with your own data. There’s the whole thing called chain of thought, which is I tell the AI first, build a series of steps, a recipe to solve the problem and then solve it, which some people say works better. Uh, that’s what they mean by reasoning models. It’s a whole charade. It’s nothing really new there. It’s more like using this chain of thought thing. But I would say you can’t really eliminate hallucinations and they happen when you least expect them. So I would not use AI in the traditional way when hallucinations matter.

Hallucinations don’t matter in some context, right? If you’re translating reviews of hotels on TripAdvisor, nobody cares if they’re a little bit hallucinated than they are. Sometimes it’s just the translation’s not very good. But I just wanna know if the hotel is clean, if it’s well located. I don’t need a perfect translation, right? But you can’t have it hallucination in literary translation, right? And probably in customer service chatbots, you can’t have hallucinations either. It happened to Air Canada that they, you know, it gave the wrong policy to a customer and then Air Canada had to honor the policy because a judge told it to. It’s like, you can’t just tell the client in the chatbot, yes, you can do this, we can get a refund for your flight, but then no, actually you can’t, right? So that’s not a thing.

So if I were to work on this, I think I would use the chatbot as a way to let people find the information they need officially. So the chatbot can be a nicer way, let’s… you could try to interpret what people say, and then point them to the right place which is the official data. And you should be very careful about not letting the chatbot, you know, interpret results directly, you know. Dunno exactly how you can do that, but I would frame it more in a way of how can I help you find information, right? That kind of thing.

Another thing like I feel like a lot of people are praising AI so much and the things that they can do, but it makes me wonder if they just weren’t doing things very well in the first place. And it happens a lot with customer service. I used to work for a company, it was a travel company. And I understood the issue with customer service, because… I never contact, you know, customer service unless something really odd happened. Otherwise I just go and try to find the information that I need without having to contact anyone. But that’s not how everyone does it. In this travel agency, people, they literally call to say, how much luggage can I bring with me on the plane? And this bombarded the customer service people. And the first thing you imagine is, I’m gonna create a chatbot for this. But I was wondering maybe the problem they had was communication. Maybe a few days before the trip, they should send an email to people telling them, hey, you know, your flight is in two days. This is the how much luggage you can bring. And maybe that will solve the issue, right? And I think there’s a bit of, you know, thinking AI can fix problems that you maybe could have fixed.

Another thing that doesn’t cease to amaze me is this whole thing of AI is so great for programming because it eliminates all this boilerplate code. And my question is, why do you have any boilerplate code in the first place? I don’t have any in my work. I use libraries, frameworks to avoid… Every time there’s anything that seems like will be a bit repetitive. I already don’t do it. I’m like, what, what library can I use where someone has done this and or how can I avoid this? I mean, some people I’ve said this before, and some people told me, oh, but look at Django. Like when you use Django, you need to, you create a new view and you need to put the URL here, this there, and that there. And I’m like, it’s literally three things. Is this what AI is really helping you with? Like I still don’t see why people have so much boilerplate repetition, right?

So sometimes I wonder is there anything deeper here that maybe you’re not doing well and now AI lets you do it better, but maybe there was a different way before, right? Um, I, at one point, I asked ChatGPT for an algorithm to calculate the position of the sun based on the time of the day and your location. And it, on repeated executions, it proposed different things. But one of them, it just provided a very long algorithm filled with numbers and trigonometry and it had constants in it like numbers. And I was like, yeah, I could maybe copy paste this in my own code. I will never know if this works, if it makes sense. But there was a much easier solution, which was like, pip install, sun position calculator, which is the name of a library that does this, right. Again, I had already, I’d already done this, right? So I wasn’t trying to get ChatGPT to do it for me. I was more like, I was curious to see what would happen. So I then I kind of executed the query again, and ChatGPT did propose a library at another attempt instead of giving me the code. But it kind of hallucinated the way in which the library was used, how you called the function. So when I tried to run it, it was, it didn’t run, right? I was like all this effort, when in reality what you can do very, you know, every person who’s coded for a while knows that there must be a Python library that calculates the sun position. This must exist ‘cause the ecosystem is huge and people have validated this. It’s a library that you can go and check the community. Are there lots of comments in the, and on the GitHub, you know, a page. And there were a few actually libraries that did the sun position, but very quickly it was okay, this one has a very nice interface to use and there are lots of, it’s an active community. So, you know, in five minutes it’s running, right?

So I don’t mean to say that AI can’t help you because it can. But I also see a lot of that, like are we forgetting about good engineering principles? Are we forgetting about good customer service? Can we understand what the most common queries are? Created a frequently asked question section. Send emails to remind people of stuff. Cause a lot of your problems can be fixed by targeting the underlying causes of the issue, not by using AI.

Henry Suryawirawan: Yeah, I was laughing when you explained that, because I think it’s such a quite insightful, right? So because people think just by applying AI, AI can do smart things that can, you know, solve all the problems, you know, seemingly, right? But I think we know, like it’s probably not possible to get a hundred percent. And I was even frustrated sometimes if I get response from like seemingly an AI, right? Uh, that tries to even first re-explain what I just express as my complaint or, you know, problem. And then it gives like some basic information that I should have known in the first place, right? And then I have to wait again. So I think it’s always very frustrating.

[00:45:49] The Important Role of Benchmarks When Building AI Products

Henry Suryawirawan: But these days people, when they build AI products, they seem to always rely on this thing called eval method or eval test, to actually, you know, kind of like train your solutions such that it doesn’t deviate so much probabilistically. So tell us about this approach. Is it something that is useful and people have to rely on? Or is it something also that feels, uh, you know, like impossible to actually come up with 100% accurate solution?

Emmanuel Maggiori: Um, I don’t know exactly that approach, what it is, but I think I’ve been saying to people for a long time, you need to, whenever you’re going to build an AI-based product, you need to create your own benchmark to measure the performance of AI at the task you’re trying to solve. There’s a whole process to do that, that is a little bit more complicated than it seems, because you do need a validation set which is separate from a test set. Your validation set you use, you use it to select among different models, among different options to pick the best one. That’s a little bit biased ‘cause you have selection bias, right? You could pick something that works really well only on that validation set. And that’s why you need the test set at the end where you perform an additional check, right? And you never go back. If you don’t like the result of the test set, you need to collect a new test set with new data. A lot of people don’t do this carefully enough and you know, it’s up to you to see how far you want to go to that process. But I’ve been telling people you do need a benchmark to start with. And then once you have your benchmark, you can see if a RAG approach works better. If a fine tuning approach works better, you can compare different models.

There’s also something pretty catastrophic that happened a few weeks ago, which is that when OpenAI launched their GPT-5, is it five? The latest one? Yeah. They removed all the previous ones, right? And a lot of people were relying on previous models. So if you’re forced into a new model, then you need the way of benchmark, benchmarking to understand if the performance has changed on the problem you’re solving.

So yes, now this will tell you how good AI is for the job. It may not be good enough or may be good enough, you know. It will help you assess, you know, the hallucinations that… I would also be careful with just counting errors, you know. You also may want to qualify them because sometimes just by counting them, you don’t realize how bad the errors can be and the impact they can have. You know, people have been saying a surgeon that is 99%, has a 99% survival rate versus an AI surgeon that has 99.5, which one would you pick and stuff? Okay, but does the AI surgeon cut the wrong leg sometimes, even if he doesn’t do it very often, you know. That’s a kind of a question.

But yeah, I think it’s a good approach to do the benchmarking to, you know, try to understand, do hallucinations matter, because they probably will still be there. Do errors matter? So do we still need a human into the loop, do we not? Lots of things like that, that I would definitely, is there another solution, like I said with customer service. Can we do something else? What’s the real problem?

Another question is, is the problem I’m trying to solve a conversational problem? Is there language involved? Because I see people do this, they’re building a travel planner where you actually input destinations from a box. You select them from a box, like a it, not a box, like a dropdown, right? Or a search box. And then what it does is suggest an itinerary in a very structured format. Under the hood, this company was trying to solve it by using an LLM. So they would take all the user’s query, transform it into a natural language problem, give it to the LLM, then the LLM returns something, you would have to parse that result to convert it into. And it was super slow, for starters, right? It was really, really slow. And I told this company, nothing of this has anything to do with language. And the easier solution that they actually found was we’re just gonna use a Google Places API, which just gives you, you know, ‘cause it was an itinerary planning to visit a place, right? Or a city. So you can actually get up-to-date opening hours, which ChatGPT can’t do. So you just go and get them from Google Places and you use a typical algorithm for like pathfinding, you know, like Dijkstra’s algorithm, which is there are libraries for that. So they kind of and that was really fast. And they solve a problem without any LLM. So LLMs are best used when actually for a chatbot, for instance, when it involves language. But a lot of people just trying to use them for everything.

Henry Suryawirawan: Yeah, I think that definitely, it’s kind of like people think it’s a holy grail, right? And in fact, I think sometimes maybe because of the hypes, right? People just creating new libraries, new tools, new stuff that works, that relies on LLM or maybe prompts, so to speak, right? They kind of like fit in, you know, the solution with these prompt engineering kind of thing, right? Try to come up with a, you know, as best prompt as possible, give as much context as possible. But you also mentioned, you highlight when the foundational model change. Definitely, things probabilistically will change, maybe sometimes by a lot, maybe sometimes not by a lot. But still you need to kind of like validate and test your approach, right? Because when this happens, probabilistically, you might give a wrong output and suggestion as well.

[00:50:57] Advice for Software Engineers to Deal With AI Concerns

Henry Suryawirawan: So I wanna move on to the next section by, you know, discussing about the concerns people have. Uh, one thing about AI, it creates a lot of possibility, but there are also a lot of concerns from people that, you know, what will happen to my job, what will happen to my role, what will happen to, you know, the world economy and all that. So maybe from the first discussion, I wanna highlight the concerns, right? Uh, especially for tech, software engineers think there are many, many software engineers that can be replaced, including juniors. What is your opinion about this?

Emmanuel Maggiori: Yeah. Well, I-I’ve written a lot about this in my second book on AI, which is called the AI Pocket Book, which we haven’t spoken about yet. But part of the goal of that book was to answer these questions. I was approached by a publisher. They thought, we need a book on surviving AI for software developers, right? And I was like, okay, I will definitely speak about the whole job market situation. I think that there will be jobs that will be lost to AI for sure, maybe prematurely sometimes. ‘Cause we’ve already seen companies firing people and now hiring them back because it didn’t work. So I think that, I mean, this is, this could be a, you know, we can speak about this from many angles, but for the angle of a person who wants to preserve their job, I would say that there are different kinds of jobs. Some jobs, AI may be able to replace, and some, it won’t. So as a professional, you want to position yourself in that latter category, right?

I think that there’s a certain level of work that requires a lot of excellence. When you go above and beyond, when you rely on a network of other people, you need interactions, you need multiple skills. In the business world, that job is oft-… In the software world, it’s often at the intersection between software and business. So you are not a person who takes a specification of a UI and codes that specification to the letter without an opinion, right? You are the person who will help the business choose the best feature to build. There are lots of positions that are like that. If you work for startups, for example, you may need to be a jack of all trades, a person helping select features. As you progress in your career, you’re closer to the business very often. So my advice is often try to work at the intersection between business and technology. See yourself as a person who helps build commercially successful products. Not just write software. Because I think you will be quite irreplaceable when you’re that person.

And another thing I think is that a lot of people speak about other people’s jobs. They say these people will lose their job, but they’ve never met a person who does that job and they don’t know what it takes to do the job, right? Especially at the excellent level, right? I’ve seen this a lot. You know, people say screenwriters will get replaced with AI or replaced by AI. But then, have they met a screenwriter? Have they seen what it actually takes to produce a show that people love, that people laugh at or laugh with, you know? I am listening to this podcast called Office Ladies, which is, by the actresses from The Office where they tell you stories behind the show and you can see how much thinking there was sometimes about certain storylines, you know, that they decided to include in the show. There’s so much that goes on, you know, in writing a show. Yes, there are some cookie cutter show, soap opera kind of show that maybe AI can do, but there’s so much more that happens when people are writing a show.

And the same thing with programming with code. I feel like maybe I’m also a bit more senior in my career, but most of my work in software is not writing code in terms of, you know, I’m gonna move this button from here to there. There’s a lot of thinking about security, about architecture. A lot of this has to do with negotiating with the business. A lot of it, you know, the business wants to do this, but this is really big and they don’t know it. So I explain to them why it’s big and then I propose a different solution. What if we do this other thing? It’s not as great, but we can do it quickly and then we can see how people react. And then we see if we build the bigger thing that you want. All of these help, you know, as part of software development, but it’s not writing lines of code.

So if I had to recommend to a software developer, again, I would say, a lot of interaction with business, a lot of high level work business, learn about business, you know, read business books. ‘Cause a lot of us, we have no idea about anything related to business, to sales, to value propositions. How to define a value proposition, for example. There’s a book called Value Proposition Design, which I, which I recommend. So yeah. I think that a good way to protect that your job is that, you know, yeah.

Henry Suryawirawan: Yeah. So I think that’s very good advice, right? Because obviously when we talk about AI replacing jobs is probably the first, will be the mundane things. You know, like producing lines of code, translating from, you know, one thing to another, or even like maybe when you say about screenwriter, maybe not necessarily screenwriter, but someone who polish, you know, some writings that, some drafts that people do, right? So those kind of things, definitely the first prospect that could be replaced, right? But if we keep moving to the higher level. Like when we talk about programming language, also the same, right, it started with like machine language then becomes general programming language. Now you have conversational language. If we can think of ourselves moving to the higher abstraction maybe, so to speak, maybe we can even amplify, you know, the value that we can produce, right? So I think I agree what you said that software engineers need to understand more about business or maybe thinking in terms of, you know, how can we make this more successful, commercial, some psychological aspect, human aspect that probably software engineers can also explore.

[00:56:49] Advice for Junior Developers

Henry Suryawirawan: So how about the juniors here? Because I think the juniors might be kind of like frustrated, right? So I just finished my study for example. But now I’m finding it hard to find a job. So do you have any advice for them?

Emmanuel Maggiori: I mean I spoke about this excellence side of things, but there are other things that will be helpful. One of them is when you work on very custom, customized work if you want, that is very kind of complicated and narrowly specified and stuff. If you work on complicated stuff, and I can give you a couple examples, those things are unlikely to be replaced by AI, because first of all, you can’t even describe to an AI what you need to do. Even the business people will not understand.

And, uh, I can give you an example. I worked on a project that involved modeling the propagation of temperature or of heat inside a building, right? And I was writing the software for this, right? All these complicated calculations, but it was so custom and they were devices connected to buildings using APIs where we get the data and then physical models. And there were parameters that define, you know, the insulation of the building. It’s very hard to, for me to even imagine how you could describe all of that in a prompt which is not the same as writing the code essentially, right? Uh, so I would say you want to specialize or to gain experience on tightly controlled kind of software.

An example of what’s not like that is, you know, building an prototype just to show something to a potential client, but it doesn’t really work. There’s a lot of MVP or no-code kind of thing going on there. It’s like we’re gonna build not a really functional thing that works with lots of clients, with lots of customer requirements, we’re just gonna build a very rough prototype to validate an idea, right? I think those are the jobs where people are gonna be replacing that with AI a lot. But then you can do very, very custom kind of like, focus on really tightly defined, custom complicated problems. You know, if they can be a bit scientific or they require mathematics, you know, it’s likely business people will not have studied those things either. So they won’t be able to know where to start with those problems.

Also working on anything that is mission critical will help you. When I say mission critical, something that you deploy a piece of software that could lose a company a lot of money if it wasn’t used correctly or it could even cause harm to people. Because this is going to require a lot of manual validation of manual input of, you know, I don’t think that’s going to be replaced by AI anytime soon. So if you can become the person who learns about those things, then you’ll be better placed to find a job or to preserve your job.

[00:59:34] Vibe Coders and Prompt Engineers: New Jobs or Just Hype?

Henry Suryawirawan: So I hope people could learn from these tips. So when you mentioned about building prototypes, right? My mind straight, straightly go to vibe coding, right? Uh, I think many people think, you know, now everyone can code, you know, just doing vibe coding and even it might be creating a perception that you can be a vibe coder, you know. There might be a job role as a vibe coder specializing in vibe coder. Same thing like prompt engineering. You know, I can be a prompt engineer that can solve, you know, maybe general purpose AI conversational kind of a thing. So maybe tell us, uh, your view about this and, and will in the future, we see new job types being created, something like this, or is it more like a hype now?

Emmanuel Maggiori: Now I think there is a place for everything and there is a place for a job title that is a vibe coder potentially, because we’ve seen that with Bubble, you know, Bubble, the no-code tool. I’ve seen people who specialize because you need to actually learn how to use those tools at the end of the day, especially if you want to build something custom. Which is kind of part of the problem here, is if you just type it a little prompt and you accept what comes out of it, then it’s fine. But if you want to start, you know, like with no-code, you often need to learn what a platform does and how to use it, how to connect it to data sources and all of that. And I’ve seen people who specialize in helping companies build with Bubble, right? Because it’s cheaper, it’s quicker, right? You’re not gonna build production-grade software. That’s the illusion that we need to kind of eliminate. And whenever you want something really custom, it’s as difficult as coding. Because coding is that. It’s literally telling the machine what you want it to do, right?

So the illusion that, that’s the same as building production-grade serious software, I think that illusion needs to go. But there’s definitely room for that other thing in between where you help with MVPs, where you help with prototypes, right? We could see UI designers maybe using AI to create better prototypes for their clients, which are more functional than before. So there will be changes definitely in there. And there will be new jobs and new opportunities, I think. Yeah.

Henry Suryawirawan: Yeah, so hopefully as with a new, I dunno, technological disruption, I think there will be a lot of fears in the very beginning. But hopefully after that we can like thrive and, you know, maybe in the new opportunities people move on to new kind of skillset and solving bigger problems and things like that. So hopefully that will be the case.

[01:01:55] The AGI Possibility

Henry Suryawirawan: And speaking about that, right? So I wanna touch on this one thing that I think it’s, uh, worth to discuss shortly, right? So the AGI possibility. So I think many AI companies think AGI will happen, I dunno, three years, five years, whatever that is. Sometimes I listen to podcasts as well, you know. There are seemingly AI experts telling that humanity is in crisis. So personally do you think AGI will happen or maybe some resemblance of it? So yeah, maybe a little bit on this part.

Emmanuel Maggiori: Look, the first thing, okay, AGI, artificial general intelligence, right? A computer that can do anything a human can do. It’s not very well defined, but we’re talking about really super powerful AI. Well, the first thing I want to say, it sells so much to make those predictions. There’s nothing that sells more. People have built entire careers around predicting that. You know, uh, an example is Ray Kurzweil who’s written these books on the singularity, saying the singularity is going to happen. Singularity is when the machines become so smart that they can improve themselves at an exponential rate and everything collapses. And it’s like like sci-fi world. It’s, people are fascinated by that and it sells so much. I feel like people who speak the way I speak about AI, and I’ve heard the same experience, you know, from other people, we actually struggle more to be heard because people don’t invite you to give talks as much. Who do you want on the stage? There’s a reasonable person telling you AI is smart, but it’s also kind of dumb. Or do you want a person telling you how it’s going to be the Terminator, you know? And it seems like the latter is, it sells a lot. And when I was looking for a publisher for my first book on AI, I remember I spoke with an agent that he’s a pretty prominent agent. Like a book agent. And he told me, no, like your book won’t sell because you have to either say that AI will be so amazing, that will change everything, or they will destroy everything. If you say something kind of in the middle, nobody really like that.

So the first thing is like this AGI thing is, and it’s also self-serving because a lot of people are making money from selling the shovels or selling the AI to build AI or AI for the sake-, and then they tell you AGI is coming, right? I would tell you the answer is no. So AGI is not coming anytime soon. And the reason is that the current methodology used to build AI, which is machine learning, has some flaws. And these flaws cause the hallucinations, for example, which will prevent AGI, right? They are not solvable right now within the methodology that we know. This means that if a new methodology, something different from the current type of machine learning were invented, maybe it could be solved. I don’t know. There’s also a question that in principle, this is possible or not. But even if we put that aside, if a new methodology is discovered, maybe. But nobody knows that methodology now, right? We don’t know it and we need to acknowledge that we don’t know it.

When GPT-5 came out - and it was a flop, a big flop, and a big disaster - cause it was nothing new. It was maybe a little bit better, but not really. I wasn’t surprised because nobody has discovered that secret ingredient or if you want that new methodology to solve hallucinations and stuff. So to me, it was no surprise that GPT-5 still hallucinated because they haven’t discovered anything that I know of in between, right? And you can’t predict scientific discoveries like that. You know, the transformer architecture was the discovery that led to LLMs or like to the current powerful LLMs. What’s the next discovery? We don’t know. We don’t know when it’s gonna happen. I always say the example of nuclear fusion based power, they’ve been telling us for decades that that’s the future. It’s around the corner. The problem is that the methodology hasn’t been found, hasn’t been discovered to do that. And they can keep putting money into it and maybe that will help, but we don’t know if it’s gonna happen or when, right? And that’s the situation with AGI, right?

I feel like a lot of the philosophical questions around AGI, they’re valid and it’s good to ask them. What would happen? Like what happens legally if… Does a robot have feelings? I like those questions. I think they’re fascinating, but they are not the reality of current machine learning. Like this thing of I’m going to learn from data how to predict the next word. It’s not going to lead us there and it’s gonna keep hallucinating. But then if a new thing is discovered, you know, maybe.

Henry Suryawirawan: Right. Pretty exciting answer, right? So I think people sometimes, uh, I don’t know, sometimes yeah, they project something that seemingly bombastic, and then yeah, it sells like what you mentioned, right? But we have seen it, with the ChatGPT-5 release, right? So it kind of like flop a little bit. So maybe, you know, the AI hype is not as great as people sold it to be, right? But obviously these next few years will be very, very exciting, unpredictable at times, right? Because there will always be new things being invented and we just have to adapt, I guess. My stance now, even now, is just, yeah, see, adapt, use them as much as possible, understand what it’s capable of and not capable of, which is also very, very important. And try to fit in you, yourself, your skillset, your value, into those landscape, right? So I think probably that’s like the most practical thing.

[01:07:23] 3 Tech Lead Wisdom

Henry Suryawirawan: So Emmanuel, thank you so much for this conversation. So I know we are a little bit over time, but I have one last question that I have to ask you. This is like a tradition in my podcast. I call this the three technical leadership wisdom. So if you think of them just like advice, uh, would you be able to share your version today?

Emmanuel Maggiori: Yeah. So I would say, look, the first one is that there’s a term called a moat. As in M-O-A-T, right? Like a protective moat. It’s also called a competitive advantage. I think a lot of people don’t understand much what this means or they’ve read some books that tell ’em this is a competitive advantage when it’s actually, it’s actually not. I think there’s a lot of confusion around the competitive advantage, and I’ve been doing a lot of research myself on the matter. What I mean with all this is that you need to understand that building a successful business is a business problem. It’s not a technology problem. It’s usually, it doesn’t, no business has failed because you use Python 3.5 instead of, you know, whatever. And the role of a business person or even the goal of a business is, even if we don’t like it, is to create a sort of monopoly. You want a moat that protects your business from competitors. You need access to some, a special way to keep your clients or to gain more clients or to access resources cheaper than anyone else. Like there needs to be a special thing. And that thing is called a moat.

And one of the reasons I tell you, I think technology leaders need to keep this in mind is that there are a few ideas going around that are terrible. One of them is this 10x thing. You build technology that is 10x better than other technology and then you’re done, your businesses. That’s not true. And the proof of that ChatGPT was definitely, I think a 10x kind of thing. And the business is doing terribly. They’re losing billions every year. And the problem is they have no moat. There’s nothing special. Having a specific type of technology doesn’t build a moat around your business. What is aggravating here is that everybody knows how ChatGPT works, right? Because it wasn’t even invented by OpenAI. It was invented by Google researchers who published the work on a scientific paper. And I’ve been saying this to people, right, a lot. And I kind of predicted this, I wrote a book called Siliconned also where I wrote about this, about OpenAI and stuff.

And then what happened now is Deepseek comes out. And then the world is like, oh my God, it works as good as, you know, OpenAI models and what’s this gonna mean? Yeah, we knew this was gonna happen because there’s so much money to be made probably in the, in AI that a lot of other entrants, competitors, copycats are going to try to build the same thing. So you do not actually build a successful business around having superior technology. You build them around, for example, network effects that keep people engaged. Or you build them around switching costs. People don’t want to leave you because they’re kind of stuck, you know. With Apple, you have all Apple ecosystems, so you don’t want to switch to a different kind of phone. Those are the things that actually make or break a business. It’s not the technology.

Um, another thing that I think is very prominent in the technology world is a search cost. Not a switching cost, a search cost. Meaning that you want to capitalize on the fact that people will find it annoying or costly to search for an alternative. I think Canva benefits from that, right? People don’t want to use anything else because they know Canva and they’re not gonna start searching for Canva, and they don’t even have opportunities to find alternatives. Whenever they want to do any simple graphic design, they just go to canva.com. But again, why I’m telling you this? Because the key to the success of this business is that the product is cheap, right? If Canva tries to charge a lot of money, this search cost starts to not protect the business anymore because people are gonna start shopping around. They’re gonna, oh, Canva is now 200 a month. Can I use something else? And business strategies about this. And I think a lot of technologists like me, we had, I didn’t learn any of this in university. I learned how to code. I learned how to create really fancy SQL statements. But that doesn’t make or break a business. It’s business side of things and the competitive advantage is super, super important. So that would be my first, uh, I dunno what you call it, my leadership something.

Second one, there’s a lot of boring stuff to be done. The most successful entrepreneurs I’ve met have built something really boring to help a boring industry do something that was super inefficient, right? And speaking of competitive advantage, I think one of the, or the moat that they had was that it was such a niche thing that it’s just, it’s actually a small, small market and people don’t want to enter the market. Once you’re there, your clients have switching costs that are really, you know, high. And I’ve seen this a lot. I’ve seen it in aviation a lot. A lot of stuff is done on paper nowadays. People literally, they fill in paper forms even today to track the maintenance of an aircraft or stuff like that. There’s so much stuff that’s done badly. And I think there’s a lot to do in the world and in that kind of boring arena. But it’s not flashy, like building the new goggles to pilot your drone or whatever. And you know, and it’s like, but actually it’s a lot of fun to work on things that people actually use and care about.

You know, there was a person who recently quit a job at a very fancy startup and joined another company that is, I can’t name it by name, but it’s a company you’ve heard of and it’s a company that builds software that is actually very useful. And I think it benefit from the sort of Canva effect that people just go and use it because it’s cheap. They’re not gonna shop for alternatives. It’s doing really well. And she was telling me that she was actually, she was a bit concerned. She was like, oh my God, I’m going from this alleged unicorn which is never happening, you know. The company has never actually turned it into a unicorn, the stock options are worthless. And then she goes into this. But it’s actually been very exciting for her to work on a product that people use, even if it’s not sexy. ‘Cause this is not a sexy product. So don’t forget about the boring stuff. So that would be my second thing.

And then the third one, which I already said, but I think I would like to go back to that, is that if you are building anything based on AI, you should acknowledge the limitations of AI from the very beginning and embed them in the product. And that’s how you’re gonna build a successful product, right? If AI hallucinates, maybe do something so that, again, you quote a passage from the text, you do not interpret it. You don’t leave room for hallucinations. Um, I’ve spoken to companies that are doing a good job in this sense, right? They are using, or they’re building an AI-based product that helps, that generally helps save time for people without pretending that it can do the job in a flawless way and people are liking the product, right? And I think that’s something you need to consider. Yeah. So those are my three.

Henry Suryawirawan: Wow. So I find it really insightful and very beautifully said as well. So the boring stuff, I think I like it the most, right? Because sometimes we are so much into the hype, right? So we chase all these hypes. But definitely there are things that are still working. Maybe it’s boring, but it’s useful and people use, and it creates value in the world. So I think thanks for reminding us those things.

So Emmanuel, if people like this conversation, they wanna reach out to you, ask you more questions or discuss about, you know, AI stuff. Is there a place where they can find you online?

Emmanuel Maggiori: Best place is LinkedIn. I usually post stuff on LinkedIn. I, yeah, that’s where I’m, that’s where I hang out. I know it’s a little bit weird. Some people will tell me, you know, why are you on LinkedIn and not on Twitter? But that’s the main place for me to communicate with people. Um, and, uh, you can also find my books to learn more about the things I have to say. And you can send me a message on LinkedIn. Yeah.

Henry Suryawirawan: Right. So I highly suggest people reading your book, especially if you want to demystify the AI hypes that is happening now. Like try to understand the power of it, the fundamentals of it, and the pitfalls where AI might fail. So thank you so much for writing those books Emmanuel, and for this conversation. So I hope you enjoy this conversation and thank you again.

Emmanuel Maggiori: Thank you.

– End –