#260 - Creator of Meta's Hack: Your AI Will Always Cheat — Here's How to Stop It - Julien Verlaguet

[00:02:03] Introduction

Henry Suryawirawan: Hello everyone. Welcome back to another new episode of the Tech Lead Journal podcast. I’m very excited today to have Julien Verlaguet, I hope I pronounce the French name correctly. He’s famous for creating the Hack programming language in Facebook back then. Right now Facebook is called Meta. So, but today we are not going to talk just on Hack, but also things that he’s working on, which is with SkipLabs, Skipper and all that. So Julien, welcome to the show. Looking forward for this conversation.

Julien Verlaguet: Thank you for having me, Henry. I’m excited about it too.

[00:02:34] How Did Julien Create the Hack Programming Language at Facebook?

Henry Suryawirawan: Right. Julien, I think we have to start in the very beginning by asking about, you know, how you came about inventing Hack, right? I think back then maybe Facebook was how big? And why did you have to come up with this Hack, which is like a flavor or improvement on top of PHP, right?

Julien Verlaguet: Yeah, so I mean, the story begins in 2011 roughly. So back then, I was working on security engineering. So security engineering was a team that was building tools to provide better security for Facebook. And one of the things it was focused on was static analysis. So we were building all sorts of programs that were analyzing the PHP codebase because most of the logic was in PHP back then. And so the goal was to analyze the PHP codebase to find security vulnerabilities. Or even if there were vulnerabilities, at least, you know, things that could be improved, right? Like where you’re doing something, eh, it could be correct but, you know, would be worth escaping there to make sure that we got this right. And so while were working on this, we realized that, we were basically, it was going to be difficult to get a little bit further in the sense that PHP is a very dynamic language and whenever you wanna analyze something statically, the main problem that you have is when you don’t have types, you lose control very quickly.

So concretely, imagine you have a piece of code. This piece of code is calling a method. And let’s say this method name is very common. Let’s say it’s called render. You know, so if you’re rendering things in — you can imagine how many components have a render function, a render method. So you don’t really know what you’re calling. And the moment you don’t know what you’re calling, you lose control over what the program is doing, and there’s really not much you can do.

And so the idea we had back then, that’s how the idea started. And one of those who pushed the most for this idea was one of my coworkers, Alok Menghrajani. The idea was we are going to build a version of PHP that is much more strict, right? And that is going to feel like a statically language. In fact, we ended up with a gradual type design, but we started with a statically typed design where we wanted something super strict. And the goal originally was to have a version of PHP that the people building libraries relative to security, cryptography, all these things would be using, right? So we would make sure that for the stuff that is sensitive from a security perspective, we would have stronger guarantees. And that’s how the project started. The company had, I don’t know, I would say in the tens of millions of lines of code I believe back then. And the number of engineers, I think we were still in the hundreds, maybe in the thousands when we started Hack. I mean, I joined, I think we had in the hundreds of engineers, and I joined in 2011.

[00:05:53] Does Static Typing Make Your Code More Secure?

Henry Suryawirawan: Wow. I think it’s quite interesting, right? The way that you started the idea. I think some people actually still dealing with dynamic languages, maybe things like JavaScript or maybe some other languages. And maybe in the first place, do you think we all need to use static typed, strong typed languages? And since you mentioned security, right, does it mean it’s more secure if it’s more static typed?

Julien Verlaguet: So that’s a tough question. So I think that the flavor of static typing I think everybody should be using, is a flavor of static typing that is very fast and optional. Because then you really don’t have an excuse. So if you are using something, imagine something like TypeScript. So the types are optional. If you don’t want them, then you don’t use them. And on top of that, imagine something that is extremely fast to respond. It will take less than 15 milliseconds to give you a result no matter what the change you’re making. If those two things are true, then you don’t really have an excuse, right, for not using it. Like if you, if it’s really in your way of shipping a feature, then, you know, just don’t use it and just, you know, do your dynamic, be your dynamic self, I was gonna say. But if you want them they’re there. And the second argument is regarding iteration speed. That’s something that you get a lot with dynamic languages. And I would say that this is more a problem of tooling than it is a problem of language, right? You can have a very fast type checker, which is what we built with Hack. So it’s challenging to build, but you can build these kind of things. And if you have this kind of tooling, then the static typing really just comes naturally. Why would you not use it?

Regarding security, I would say it’s going to be more work to be secure. It doesn’t mean you cannot be secure, but it’s going to require more work. Mostly because in a language that has static typing, you can encode a lot of good practices in the types. So just to give you an example, imagine you have a string that comes from a user. If you are in a statically typed language, it’s very easy to craft a type for this kind of string, right? And to make sure that the stuff you’re doing with it is actually safe. That whenever you embed it into another script, string, there’s some kind of escaping going on, etc, etc, right? And if you don’t have those, then this is going to require a lot more discipline, right? To get the security right. Now it doesn’t mean that if you’re using a statically typed language, your code is secure. It doesn’t mean that at all, right? It’s just that you, it’s one step in the right direction.

But for me, the main advantage of static typing, and one that doesn’t, that gradual typing doesn’t really give you, is refactoring. And that’s a key part of what I’m working on now because now I’m more working with AIs and building tools for AI. And one of the things that you care about, well, you care about as a developer but even more so for AI is how do I refactor something? How do I make sure refactoring was complete and all the changes I had in mind were taken into account? If you have the right tooling and the right language, you can have the tooling drive the changes and let the user know what needs to be modified. So that’s I think one of the key benefits of static typing.

[00:09:54] How Did You Convince Facebook Engineers to Adopt Hack at Scale?

Henry Suryawirawan: Yeah, you, I think, mentioned about tooling, the importance of tooling. I kind of like agree as well, right? So I think when you have a good tools, I think statics or strongly typed languages, for example, I think will help you a lot, especially things like refactoring, making sure that the inputs kind of like safe according to the types that you expect it to be. And I would imagine like many people who actually like dynamic languages would hate strongly typed languages. And I assume back then when you introduce Hack to the, you know, Facebook engineers, they were not happy about it. So maybe tell us a story how do you convince the whole company, the whole engineers to actually adopt it because I’m sure there are a lot of, you know, challenges there.

Julien Verlaguet: Yeah, so, first of all, most of that work was not done by me. So the one who was the biggest advocate for Hack in the company for with engineers was Alok Menghrajani. And then the biggest advocate for managers and getting the management structure to adopt this was, Joel Pobar. I was more on the technical side and getting this thing shipped and working. But I think Facebook was not really a PHP shop back then. I think what the website, the original website started in PHP just because of the time when it started, you know, it started in early 2000s. And in early 2000s, what are you gonna write your website in, right? So you had very few options, right? You could write it in Perl with CGI or in PHP, right? I think Ruby, I’m not sure Ruby on Rails was out. So I think everybody understood that this was kind of a legacy thing that, you know, the business logic was in PHP. And there was some kind of, I would say, healthy relationship towards PHP from programmers at Facebook back then, which was, look, it’s not ideal, but we can still get our, the job done if we use this PHP and we don’t, you know, endlessly complain about the problems that the language have. But there was definitely a will to improve the language, improve the tooling. I mean, HHVM was well underway back then. It wasn’t shipped in production yet, but it was close to it. So Facebook was already investing its own version of PHP with HHVM.

And so when it comes to adoption, the fears — so there was, it’s true, a few people who really liked PHP and didn’t wanna change their way. That happens all the time, right? But not that many. I would say most of the fear were around, are we going to be able to make that con— that, the conversion of the codebase a success? And are we not going to end up with a very, you know, Frankenstein of a programming language that only applies to some parts of the codebase, right? Because back then TypeScript did not exist. You know, like nobody had really placated a type system on top of a dynamic language like that before. I mean, I’m sure some did in academia and in smaller scales, but not at this scale, right? No company had tried this kind of migration before. And so most of the concerns were more around that, right? Like once we embark on this journey with Hack, well, it’s going to be difficult to, you know, get out of it. And are we, you know, doing the right thing by investing in this thing. And so we had to invest a lot to make it really easy to get out of Hack. So, we built a bunch of tools showing that if you didn’t like the type checker, it was really easy to just strip out the type annotations and go back to a dynamic world. So we did build things to show that, to minimize the risk of adoption of Hack. That, yes.

And then another concern that we had was people who were worried about the productivity of developers. And the thing is, I think a lot of it came from bad experiences with C++. So C++ has esoteric, well now it’s much better with Clang, but back then with GCC, it had esoteric error messages, especially regarding generics. So they call them templates. It’s the same idea. And given that Hack had also introduced generics in PHP, there was a, you know, quite a few people who had a knee jerk reaction to this, you know. And they were like, whoa, you’re bringing back the, all the stuff I hated in C++ back in PHP, I don’t want that. Leave me alone. I’m very productive with my PHP. I don’t want to enter this new world of pain. So we had to work very hard on the error messages and making sure that, you know, the experience was nothing like C++ for somebody who is using the type system. But overall, it went pretty well.

We automated also a lot of the conversion. And so a lot of the pain of adding the types was not done by humans, so that helped. But another challenge that we faced was the fact that at the beginning there was very few type annotations because the type system was still new. And so a lot of the code was not annotated with types yet. And when an annotation is missing, the type system basically considers the type to be like any in TypeScript, meaning you can do whatever you want with it. And because of that, a lot of people at first thought that Hack was a linter. Because they would turn it on and sometimes it would give errors, but sometimes no, right? And it looked like a linter in the sense that it looked like a best effort approach. And really it wasn’t like if you annotated all the types, then you had an experience that was very similar to the one you would have with a statically typed language. But since most of the code was not fully annotated, that meant that anytime you interacted with something, chances were high that the type checker was not picking up the type and was not checking anything with the object. And so that was something that I struggled with for, I would say, yeah, the first six months, maybe the first year, where I was banging my fist on the table saying, this is not a linter. You need to add the type annotations. If you don’t add them, it doesn’t have any value. But yeah, after enough, you know, communicating, messaging and, you know, the message went through.

[00:17:15] How Can Engineers Overcome Skepticism Toward AI Coding Tools?

Henry Suryawirawan: Yeah, So I’m sure it was a very tough moment back then, right? I would imagine if let’s say the current AI capability is available back then, probably it will make your job much, much easier, right?

So let’s switch gear a bit to the AI related stuff, right? So I think many engineers would have benefited from using AI. But in some companies, I assume also some engineers actually kinda like resist the AI, you know, kind of tools and, you know, all these LLM, which maybe initially they were afraid because of the hallucination and giving, you know, bad code. Maybe borrowing from your experience introducing Hack, you know, for people to adopt new things, right? What would you advise engineers to do if let’s say they still are kind of like skeptical with the AI stuff that is currently going on?

Julien Verlaguet: So, I think you have to consider it for what it is. You have to think about it as a tool, right? And no tool is perfect, right? Like so imagine using a programming language or a type system and imagine I’m able to express an idea in this type system. Do I just throw it away just to say, all right, all this stuff is useless. I’m done with programming, you know, and close my computer and — No, you don’t do that, right? Like those tools, they have — they are very powerful. They can do a lot of things but they also have weaknesses, right? And so that’s exactly how you should approach them.

So what you should not do, and I think it’s going to be a problem with people who are more junior, is take an LLM’s output and just press okay, okay, okay, and ship that to production, right? If you do that, you’re gonna have problems. And the problem is the AI will provide answers, but most of the time those answers are not gonna be perfect, right? And so it’s your job to — So unless you are in a context where none of this matters, right? Maybe this is a script you’re running one time for a demo and this script is running within a VM that is well self-contained, well maybe you don’t care. Maybe you say, all right, just run the thing, see what’s happening. But most of the time it’s a bad idea to take an output and just, you know, run it in production without thinking about what’s going on.

And I think it’s harder for junior , because if LLM had existed when I was younger, I think I would’ve used them too much, right? Because when you’re young, you and most programmers are lazy, and I’m lazy myself. That’s why I like programming because I get the machine to do things for me, right? This is awesome, right? So when you’re younger, you don’t, I, you don’t yet have the experience of all the things that can go wrong, especially down the line after a long time. And so it’s very tempting to see this thing that seems to be producing things that seem to work, and just take them as they are and just ship what you have, right?

So I would say my advice to developers is number one, don’t resist LLMs or don’t resist AI. It’s going to happen. It’s already happening whether you like it or not. Second thing is, there’s still going to be room for programming but also coding. I think a lot of the coding is going away, but the coding that’s going away is a good thing. It was boring coding, right? Like the boring business logic that nobody wants to write, right? It’s a good thing that this stuff is going away. And the stuff that’s gonna stay is the interesting stuff, right? Like this super highly optimized, you know, compiler pass that has to do exactly the right thing or this very precise, you know, concurrency, you know, construction that involves atomics and whatnot, where everybody has to think very hard about every single line or this advanced algorithm that does, you know. So this stuff is gonna stay in the hands of developers for a while. Like maybe, you know, LLMs will be able to code that too someday, but we are not there yet.

So I would say LLMs are going to be used for a lot of the coding, but the programming stays, right? Like the design and making sure that things are structured the right way. This is all programming. Programming was never about typing things with your fingers, right? If it was, then, you know, I have been using way too many Emacs macros, right? Like I had a macro for everything to just type less, right? And so you can see the LLM as a super powerful Emacs macro if you want, right?

So number one advice, don’t fight it, it’s coming. Number two, use it for what it is. It’s a tool. It’s good at many things and it will allow you to move a lot faster in your job, if you are wary about the fact that the output should not be trusted.

[00:22:44] Should Junior Engineers Trust AI-Generated Code?

Henry Suryawirawan: Yeah, So I think it’s very tempting, definitely especially as the model gets more powerful. I mean, these days even like, I don’t know, with Opus 4.6, 4.7, and now with GPT-5.5, it seems like every few weeks or months we get a new leap of capability, right? And I think it’s very tempting for any engineer to just let the agents do the stuff for us.

And speaking about, which also in some companies, they kind of like have a KPI or force the engineers to just use LLM as much as possible, right? Increasing, they assume it’ll increase the productivity. So coming back to the juniors that you mentioned, right, obviously the temptation is there because we are kind of like lazy. We wanna like produce a lot of things faster. And there’s also pressure from company for us to actually use AI. How would the engineer use it more wisely? Because again, like maybe their knowledge and experience doesn’t warrant them to actually analyze what is the best kinda like piece of code that they should produce.

Julien Verlaguet: Yeah, so that’s a great question. So when it comes to company pressure, I don’t think they want you to use LLMs just to use LLMs, use AI just to use AI. I think that they know that a lot of engineers are reluctant to change because we’re just human and, you know, humans are reluctant to change. And so, you know, people missed the good old days where everything was typed with Emacs and Vim, and where that was actually an important debate, right? You’re saying Emacs or Vim? And now it feels like ages ago when really it was couple of years ago.

No, my advice — So first of all, I think it’s tough for junior engineers, but I also believe they’re probably gonna come out ahead in this game. Because the junior engineers will get to build all their professional career with LLMs when we are discovering them very late. And they’re going to develop intuitions that we don’t have now. So I think the kids are gonna be okay, if you will. So I’m not worried about them. I’m worried about certain things that can hurt them, which is if they believe too much in the outputs of the LLM, then that could be dangerous.

So for juniors, I would advise them to try to do things themselves first, especially when they’re in college and they’re not trying to ship something, you know, for a company. Getting to build the intuition themselves. And then whenever they use an LLM, my advice to them would be if, you know, somebody came around and just remove that diff and said, okay, you have to rewrite it, can you rewrite it? And if the answer’s no or if you have a doubt, then you probably shouldn’t be shipping what this LLM has written, right? So my advice to them is have a level of understanding that is as good as possible and never forget that the LLM has blind spots. And even if the look, the code looks correct, it can introduce very subtle bugs, you know? So let let me give a concrete example, right, where I got caught by this problem myself. So, I was, this was maybe, you know, six months ago, maybe a year ago, I don’t remember. But I wasn’t as familiar with LLMs as I am now. Like I’ve, I had used, you know, Claude Code, I’m using most of the time Claude Code for coding, sometimes Codex. And I was trying to get a big file to be split into smaller pieces. And this was actually an important file where I wanted the LLMs to help us. And the LLM was constantly complaining that it was too big to analyze in one go. So I was like, okay, I’m gonna split this file into smaller pieces. And therefore every time the LLM wants to help, it will be able to iterate on a smaller file and use less tokens, right? So very, very simple diff, I give very simple instructions and I say, hey, but this is an important file. This is the type checker for the new system that we built with Skipper. And very, very subtle stuff. Everything is important in there. And so I say, alright, split this code into pieces, right? It has different methods, different functions, different classes. Just make sure that you regroup them in a way that makes sense. And so I cut out the diff, I didn’t review what the LLM wrote. I’ll be honest with you, I was like splitting some file, the tests are passing, it’s just working, right?

And so I published, I put the PR up and a person on my team who’s reviewing the PR says, hey, so were you supposed to split the code or also modify it? And I’m like, no, just split. I never give instructions to change anything. And in fact, as it was splitting the code, it introduced a very, very, very subtle mistake. It replaced a union with an intersection or the other way around, I don’t remember. And it was in a place on a code structure that is very rarely used. And in our tests, the intersection and the union had the same value because we only tested with, I think it was a number or string, because this is not a very often used construction. And so, that was it. I saw this thing. I was like, but I never told the LLM to change the code, you know? And so that is typical of the kind of things that an LLM can do that a human being would’ve never done. A human being would’ve never done that because a human being would’ve used tools to copy paste and would’ve never retyped the text, you know, themselves. No human being would’ve done that. So that’s my advice.

Remember that these things are not human. And you give instructions that sound clear to you, but those instructions could be violated in ways that you don’t, that you cannot foresee because it’s not a human being.

Henry Suryawirawan: Yeah. Thanks for sharing such a very interesting case, right? Because I would imagine by now many engineers would just assume, you know, yes, yes, yes, yes. They look at the diff, looks good overall, and then they just, you know, ship it, right? So I think that’s a very good lesson. And we still need to keep kind of like remind ourselves that these LLM works by kind of like probabilistic manner, right? And that’s like the generative part of it, which sometimes, you know, they try to be creative and come up with the output that is not expected.

[00:29:44] How Do You Build Reliable Guardrails for LLM-Generated Code?

Henry Suryawirawan: Which brings us to the next discussion, which I think you kinda like pointed out as well in some other articles, right? It’s about building guardrails for this kind of behavior. I think many of the current practice of, you know, putting guardrails is to do it in the prompts where we see it in many, you know, agentic kind of like tools and behavior, right? Maybe like Claude Code, maybe OpenClaw. And maybe some other things that we build agent based kinda like software, that we built is to put it as a prompt. Just like what you mentioned, just split the file, don’t change anything. So that’s like a very typical guardrail. But obviously things fail because of the nature of the LLM. So tell us what is the best way to actually put a proper guardrail such that the AI and LLM will not be able to produce incorrect output.

Julien Verlaguet: That’s a very interesting question, and this is my new passion now. So I started working with LLMs, one, one and a half years ago seriously, I would say. And I’m very — so first of all I’m very excited about this new space, which is building guardrails and tools for LLMs because it’s — So as I said earlier with Hack, I was more the person behind the technical aspects of Hack. So getting Hack to ship and whatnot. But the human aspect is something that I dreaded and I was not doing myself. And it is good that, you know, people like Alok and Joel Pobar.  Alok Menghrajani, Joel Pobar were there to make sure that there was adoption and people were, you know, understanding what we were doing and so. And it’s just to say that this part of the job was not my cup of tea. Like I prefer to be in a cave alone and go with my computer. And so now I have this new toy where my users are computers and the humans are outta the loop finally, right? I can just focus on building tools and then I don’t have — because the complaints that you get from human beings who are using programming languages, frankly, it’s just funny, right?

Like you’ve got people who show up and who are like, you know, I don’t know that life is worth being lived without that feature. I don’t know that I can do it. Julien, please. And you’re like, dude, you’re gonna be okay, right? Like most languages don’t have this feature. I use a programming language that doesn’t have this feature. Every day you’ll be just fine. And you know, they can be very dramatic because they have an emotional attachment to their tools. And especially the programming language. And programming language is this thing that every programmer has an opinion about, right? And sometimes not as well informed as they think it is, right? It’s because it’s emotional. So it’s like, you know, asking people why they like this or that football team, you know. Like the answer is gonna be, “Because they’re awesome!” And really what’s going on is they adopted a programming language. They remembered all the good time they had with it and all the great stuff they built with it, and how productive they were. And all those emotions come in, and now you have a guy who’s criticizing it and say, no, this. Yeah, that’s going to, that’s not going to, that’s not going to be an easy conversation, right? So with that being said, I’m very happy to step into the worlds of LLMs because now humans are finally out of the loop. I can design type systems, tools, probably languages at some point. I don’t think it’s the right time to build a programming language for an LLM quite yet, but we might get there. And, but yeah, so and now I get to build tools for LLMs that where whether human is outta the loop.

So to answer your question, so how do you structure things with an LLM? So there are quite a few parts. So I think the first part is to build things in a way that are spec-driven. Because when you have a constant — so you have to remember that LLMs lose track of things. So they have a certain window, and then after a certain size, they will compress the history. And because they compress the history, they’re going to forget things. And so it’s very important to have a plan ahead of time. And then have a document that is a source of truth for this plan. So we can always go back to it and say, no, this is not what was written in the plan, right? So you need a plan that is a source of truth. Then once you’ve done that, you need to split up the work into smaller pieces. Those pieces have to be independent, as independent as possible, and they need to be more than just code. There also needs to be validation and validation against the original intent. So what you end up with is really something where you have a main document that is gonna be a source of truth that you split up into smaller tasks. Those tasks are split up into smaller sub tasks and so on, and so forth. And every step of the way you have validation for all the environments that you care about.

Now how you get there, well, there are different components. One of them is going to be the tooling. So we at SkipLabs, we build tools specifically for the purpose of helping LLMs build better code. So we re-wrote a version of TypeScript from scratch, that is both sound and incremental. So the soundness is important because we wanted to be able to build more analysis on top of TypeScript. And if your type system is not sound, that means you cannot trust the types. If you cannot trust the types, there are all sorts of analysis down the road that you cannot do, right? So the first key element is soundness. The second one is incrementality. The AI will need to iterate. And it will need to iterate really fast because it’s constantly producing new outputs. And the last thing you want is, you know, a type system that takes five seconds to re-check everything, every time you get something, right? Because that’s gonna slow down the whole process. So you want something very incremental. And so we build that too.

So this I believe, what I just said is true for any system that wants to use an LLM and have correct results. So I think anybody who wants to have good results with an LLM where they’re not constantly handholding the LLM, because that’s another option, right? Get the LLM to produce a diff, watch very, very, very carefully what it’s doing, and then if it’s doing, if you are okay with it, then say, okay, press okay, everything works. That’s what I do every day when I use Claude Code, and I’m a big fan of Claude Code. Don’t get me wrong. I love this stuff. I use it all the time. In the context of, let’s say you want a closed loop system, let’s say you want a system where you want a spec and you want the LLM to produce something correct, I think that’s the way to go. You build a spec, then build a graph of, you know, dependencies of tasks and subtasks and whatnot, and validation layers at every step of the way, and integration layers every step of the way. So every time it produces something, validate it, then take the results. Make sure it integrates with a bigger thing, validate that again with guardrails everywhere, right? And you’ll, you’re gonna need tooling to do that well. So that’s, I think anybody who wants to build this kind of system, that’s what they should focus on.

Now, I have a personal conviction that it will need to go further than that, but that’s just me because, you know, I don’t have a crystal ball. I don’t know where things are going, but I believe that latency especially of the produced code and of the harness, is going to be the key factor, you know, in the next five years. Because LLMs are not going anywhere and they’re gonna produce more and more code, faster and faster. Like as the hardware gets better, as the inference gets better, as everything gets better, it’s going to pump out code at a pace that, you know, is won’t…, you know? And so if you want the tooling to keep up with that pace, you will need all the code that is generated by an AI to be incremental.

So let me give you a concrete example. Imagine two companies. You have one company in five years from now, and they have a service that’s live, multiple services, they have an architecture. And everytime an LLM produces a diff, this diff is validated, et cetera, and we know that it’s correct. Shipping it into prod just involves, you know, takes a hundred milliseconds. Because when you ship it to prod, we took your diff, we analyzed it statically. Because we analyze it statically, we know which parts of the codebases are affected by this change through reachability analysis. Once we’ve done the reachability analysis, we have a reactive runtime. So a runtime that’s capable of invalidating the right caches, the right things, and take into account this change, and put the states of the system in a state that makes sense semantically. And a state that makes sense is basically the state you would’ve been had you restarted the program from scratch, right? Because you have systems that let you change code on the fly, like hot loading. But the problem is nobody really knows what state you’re in. You’re like, okay, now this function does a different thing, but what’s the state of my program? How do I debug that? That’s very unclear.

So changing the code is one thing, but that’s not the important part. The important part is updating the state of the system so that the state is in a state that makes sense. And the only state that makes sense is had I restarted everything from scratch, would I be in this state? If the answer’s yes, that’s the state that makes sense semantically. So now imagine company A has an AI that produces diffs, and whenever a diff is produced and validated, it’s in production in a hundred milliseconds. And now we can have hundreds of AIs constantly updating the system, making it better, self updating, fixing bugs on the fly. Company B still has the tooling from today, right? So company B is, puts up a PR, the PR spins off CircleCI, Circle — I mean, not that I don’t like Circle, I love CircleCI. I use it every day, don’t get me wrong, right? But you get the idea. And then, you know, we re-run all the unit tests from scratch, takes half an hour, and two hours later maybe you have a different production, right? And then you have to roll it out in stages and you know the drill, right? So that is not the future. I think as AI produce more and more code, we need to build the tools so that they, we can keep on shipping at this pace. So that’s where, what we investing in at SkipLabs.

Henry Suryawirawan: Wow. Very exciting, the kind of like future that you are foreseeing, right? And I kind of like agree to some extent that, you know, everyone now is easily producing code, right? Churning out PRs over PRs over PRs. And I think the bottleneck just shifted, you know, from actually writing the feature into, you know, building, validating, deploying, right? I think, you know, when you have a CI that always runs, like maybe building container and, you know, verifying all the code, compiling it, I mean, it takes time, like probably if a company that can do it below one hour, I think that’s already great, right? But I think one hour is also like kind of like a long time for a small diff. So I think I’m very excited to hear more about that thing.

[00:42:15] What Validation Strategies Prevent AI Agents From Cheating on Tests?

Henry Suryawirawan: Maybe if I can ask a little bit more. So you’re advocating people now to do spec-driven development, at least like come up with a plan, break it down to kind of like sub-tasks and put validation. So maybe my question is about the validation. Well, what kind of validations do you think will be good? Is it like tests, unit tests, or is it like linter or maybe some other things that you would advocate people to put validation as part of the plan?

Julien Verlaguet: All the things we’re doing today, but I think we’ll have to adapt them for AI. So there’s definitely tests. There’s also typing, linting, all that stuff. We are building other kind of analysis at SkipLabs. We started working on an abstract interpreter, for example, that can give you properties over abstract domains in the program. And those are interesting.

But I think when you put them in place, you should be careful about how you put them in place. And remember that an LLM that you are working with. And for example, when you’re writing tests, remember that the LLM is always going to try to cheat. And it’s true. It does cheat. So it will try to fix the test one, two times, and then eventually it will just make the test say what it wants to say, right? So you need guardrails there too. You need to make sure that so — Okay, so let’s say you are not sure that the test is correct, right? So if that’s the case, ask an AI to deliver an opinion However, don’t make this same AI fix the test. And don’t let it, you know. The AI who you’re going to ask to, is this test correct? Like is this code supposed to do that, should have no other context and no other focus than that. That’s the sole job of this AI is to say yes or no. The moment it said no, we’ll fix it. If it said yes, then you need another AI. And so now let’s say you want an AI to fix the code. You need to make sure this other AI that’s going to fix the code cannot touch the test.

Another thing you need to make sure of is the code that is produced is actually calling what you’re trying to test. ‘Cause I’ve seen an AI doing that, right? So there was a test and it was saying, oh, you have to return this value. And it was iterating, iterating, iterating, clearly could not produce the value. And it just made it return the right value, you know, and without coding the thing it was supposed to test at all. And so that’s kind of thing where you need tooling. You need tooling that’s going to analyze the code. And if the code does not reach into code that’s supposed to be called code, then you have a problem. You also need runtime checks because I could write something like if false, call into the code that’s supposed to be checked. And the AI is gonna figure out all this stuff, right? So you, the — yes you need to do all the things you would’ve done with a human, you know. Like type checking, linting rules, unit tests, integration tests, regression tests, all that stuff needs to be in, but it has to have an AI flavor, right? You need to remember that this is not a human being. It will try to cheat all the time. So guardrails, guardrails, guardrails, all the way.

[00:45:54] What Is Skipper and How Does a Closed-Loop Coding Agent Work?

Henry Suryawirawan: Yeah. And also don’t forget human in the loop sometimes. Yeah, you really need a human to help verify. Although with more and more outputs produced by AI, I’m sure it’s also not possible for humans to look at them.

So maybe if you can share a little bit more the work that you did, you mentioned just now you kind of like build a new kind of like flavor of TypeScript. Is it ready for people to try? Is it backward compatible with existing TypeScript? Can people try it out? Maybe a little bit more about, you know, what you are building at SkipLabs.

Julien Verlaguet: So we are building Skipper. So Skipper is a coding agent. This is closed loop coding agent. So it’s something where you — it works a bit like a compiler, if you will. So you give a spec and we are going to produce a program that works, and it’s not something that you’re supposed to chat with or like Claude or Codex or all these other coding tools. It’s really supposed to go all the way down to something that works, right? So the key thing is that you have to give up control over the code. So you can see the code, but we won’t let you modify it or chat with the AI as it’s producing the code. So it really works like a compiler. You say, I want this, and it will produce something that works, right? So, and in order to do that, we built quite a few things. So I would say there’s two aspects to it.

One is a development environment. So as I said, we built a type system from scratch, which we call SKJS. So it’s fully compatible with TypeScript, meaning it has the same syntax, doesn’t introduce its own constructions or anything like that. So a TypeScript program is also an SKJS program. But the types system has been rewritten to be sound and it’s also incremental. Now it is not meant to be consumed by humans, and I think it would be painful for humans. Like maybe one day we could release it if we worked on it a long time. But it’s not for humans because it’s the contrary of what humans want. What humans want is something flexible, something easy to use. This thing is not easy. It’s going to corner you into doing things in a certain way. Now it’s trying to stay as idiomatic as possible so that the output from LLMs, the LLMs don’t have to get outta their way to match the spec. But on certain things, especially with, for example, dependencies and states, it’s going to be very hard. And mutability. So basically the type system wants to enforce the fact that we have control over what you depend on and the states that you introduce or not. So every time you introduce states, we want full control over the state because we want to understand, you know. A state is one of the hardest things to manage in a program. And so that’s why we are so careful with state.

So I don’t think any human being would wanna use that. If they want, they should contact us. But that’s not the goal. The goal of this type system is to be consumed by an AI. So we built a development environment. There’s a type system. There’s also a reachability analysis, which I mentioned earlier. So this is something that, given a change, we are able to determine which parts of the code are affected by this change. There’s also an abstract interpreter that we started working on, so I don’t know if people are familiar with the concept, but an abstract interpreter consists — So you can think of it as a normal interpreter, except that the values are domains, abstract domains.

So think of it in your head, imagine you have a JavaScript interpreter, and when it’s running your code, when it sees a number, one, and it adds two to it, it does one plus two, okay? Now imagine you change those values one and two, you change them with abstract domain. So for example, you say, I don’t know that it’s one, but I know it’s the interval 0 to 5, and one could be part of it. So now the values of your interpreter are these abstract domains, right? And so when you run your code, when you do zero five interval added to two five, well, the result is going to be an abstract domain. The result is going to be to, you know, whatever the other number was. But you get the idea.

And so what’s nice with these kind of approach is that you can analyze the code and find properties about the code that, without running it, that you know are going to be true for the different code paths. And we wanna use those results to guide the AI into producing better results and produce better code.

The last aspect of the development environment is a reactive framework. So what we have is, we need at runtime to be able to iterate on the code very fast. And the last thing we want is restart everything from scratch and rerun everything. And so to not have that, you need a way to structure your code, to structure your runtime that allows that. So this is the development environment, three pieces: a type checker, reachability analysis, and you know, abstract interpretation, and then the runtime.

And so what happens when a diff comes in? When a diff comes in, what you do is you first type check it, but it’s very fast because the type checker is incremental. Once you have it, then you can run, once it passes the type checking, you know the type of everything so now you can do a reachability analysis, so you can know what is affected, what is this change affecting. And then, the runtime, so the program that is running, you don’t need to kill it and restart it. You can directly apply the change and update the state of that program. So that gives you a loop that is just incredibly fast. The LLM will be able to iterate at a speed that is unbeatable. So that’s for the development environment.

And for the harness, which is the other part. So the harness is basically what I explained earlier. So you take a spec, you break it down into independent pieces, then tasks, sub-task, et cetera. And you make sure that everything is tested and validated, and that different loops of different sizes. So you can have a loop where you’re iterating on a very, very small piece, but then once you try to integrate them, you have a bigger loop that makes sure that this is integrated, et cetera, et cetera.

The one big change though is that we, the harness and the code that we produce in the harness abstracts away state and concurrency. So we adopted a programming framework, a programming model, where the model never has to worry about states updates and never have to worry about cache Invalidation, never has to worry about concurrency, because all of that is abstracted away. So the AI builds a graph of computation, and then the runtime runs this graph of computation runs the updates, takes care of the concurrency, takes care of the cache invalidation. And so the AI can really focus on the logic, here are some inputs, produce me some outputs, and we’ll validate every, everything, every step of the way.

So two parts: A development environment with pieces that makes it all incremental, type checking and all that stuff. Second parts of the harness that breaks up the into smaller pieces and that produces code that abstracts away, I would say everything that’s hard. Not everything that’s hard, but things that are very hard about computing, right? State management, cache invalidation, this kind of stuff, concurrency.

Henry Suryawirawan: Yeah, I was about to say, those are the hardest thing of, you know, programming, right? So When you need to do cache invalidation, you know, concurrency especially, you need like, kind of like abstract, you know, perspective in your mind. But obviously the one thing that I am also quite excited is the hot reload thing that you mentioned, right?.

I’m sure every maybe TypeScript, JavaScript developer, has experienced some form of shape, this kind of hot reload as your code gets bigger and bigger. It used to be very slow, like whenever you’ve changed something and yeah, you have to wait few seconds even on your laptop, right. To actually see the changes and sometimes even it invalidates the whole state again so that you have to redo whatever that you do, you know, during the development.

[00:54:59] How Does Skipper Compare to Claude Code in Terms of Correctness?

Henry Suryawirawan: So I think very excited about that. So also the second thing is about the harness. You actually also mentioned it, in one of the article that if you compare your harness with Claude Code, it can actually produce much better correctness. So tell us about this approach. How do you actually compare with Claude Code? And maybe this also comes back, how do you actually verify the output? Are the output actually the same or actually in different form?

Julien Verlaguet: So yeah, so first of all, Claude Code is awesome. It does a lot of things and it does a lot more than Skipper. So I don’t wanna say bad things about Claude Code. I use it every day. I think it’s a beautiful piece of technology. So with that being said, the one thing that we are better than Claude Code is shipping code where there’s no inter- human intervention whatsoever. So what we’ve done basically is we’ve taken scenarios, so prompts that are non-trivial, right, that describe a program, right? Typically a service, right, that says, you know, you need to have this input that, or that output, this value, that value. And you basically explain what you want to produce. And then what we’ve done is we ask Claude Code to implement this. And we give, yeah, and that’s it.

We give the prompt and say, produce something that works. And then we compare it to Skipper. And so what happens with Claude Code is that it’s not very good. It doesn’t give very good results without some amount of handholding. So it’s, so oftentimes the biggest problem is that it says that it’s done when it’s not. And so what I mean by done is once the program is produced, then we have tests that verify that the program is actually doing what it’s supposed to do. And so the number one problem is that Claude does not implement everything. Or, you know, says that it’s finished way before it’s actually finished, right? And then, but then if you iterate with it, eventually it will get to the right solution. But that’s what we are measuring. And so our stack, so Skipper is a closed loop system. So the downside of a closed loop system is that we ask you to hand over the keys of your code, which is a huge ask, right?

Like most developers are not gonna go for it. And it’s not usable in every context, right? Like many developers are gonna say, no, I want my codebase, I love my way of doing things, I, okay. But if you’re willing to do that, let’s say you want, you know, a service, either an agent or a backend service or anything, you know, that’s running. And you don’t care about the code, you just want it to work. Then Skipper is a much, much better solution than Claude Code as of today. Maybe they’ll launch something better in two weeks. Who knows? But Skipper is a great solution for that. You write a prompt in a file so you, it’s not back and forth. It’s like a compiler, right? So I write a file, a prompt.md file that describes what I want. I run Skipper. You let it churn. And it comes out with something that works.

[00:58:27] How Do You Get Started With Skipper and What Does the Output Look Like?

Henry Suryawirawan: Wow, sounds very interesting! Like if you don’t allow the programer to actually touch the code. I kind of like still remember maybe the analogy just like the punch card back then. You kind of like do something and then you plug it in and see the output, it’s fine or not, and you change, you know, the specification again. So very excited. So for people who want to try it out, right, where do they go? And again, like my question also like, when you say people cannot touch the code, does it mean that the code output is different, it’s not JavaScript or it’s not, you know, TypeScript or something like that?

Julien Verlaguet: No, it is TypeScript, and the code output is TypeScript, and people will be able to read that TypeScript and run it with very, you know, vanilla tooling. So — and the license of everything involved in that thing is very permissive, so the code is accessible. It’s just that it’s going to be very, very difficult for a human being to intervene on that code. So what’s fine is if a human being wants to say, you know what? I wanna take it on from now. I wanna take this code and maintain it myself. Now this is totally fine. You can do that. But the moment a human being wants to intervene on the code, it’s going to break the environments that we have in place. So we would have to expose all the tooling and all the harness and all the logic that we have, that checks that, you know, the modifications are what we want them to be. And I think it’s going to be way too constraining for a human being. I think just the type checking part, which is the tip of the iceberg, is already very constraining. You cannot use the libraries you want, you cannot manage states the way you want. You cannot, I mean, it’s not your typical JavaScript project, right? So I think most developers are going to give up on that. And we chose TypeScript specifically because AIs are very good at generating TypeScript. In fact, in our evaluation, so the choice was Python or TypeScript because they’re very good at Python too. But we, TypeScript was easier for us because there’s a typing base. And so we didn’t really have to reinvent a type system. I mean, I know there’s been attempts in Python, but they’re not as adopted as TypeScript. And the other reason why we chose TypeScript is because there’s a JIT, and so the code is faster than Python. But yeah, I mean maybe one day we’ll make a version where humans can intervene on the code. I don’t know. But it’s, it would be a lot of work. It’s not ready now. And you will be able to use Skip next week. We are launching it next Tuesday. So very excited about that.

Henry Suryawirawan: So maybe in anticipation, because by the time this podcast is released, most probably the project is there. How would people get started? You know, like tell us, like should they start with an empty, you know, new service or can they start with existing projects? So tell us a bit more tips, like how can people get started.

Julien Verlaguet: So the way they start is think of a service. So first of all, I say service, but the fact that it’s a service is a detail. Because the way the program is structured doesn’t really matter, right? Like anything can be a service. A compiler can be a service, right? So it has to be something that they’re willing to restart from scratch with an AI and completely hand over the control over to the AI. So imagine you have, you know, an agent that you wanna write and, or you want an agent that reads your emails and do things with it and whatnot. You write it down in a prompt, you create a directory, you put this prompt in it, you invoke Skipper, and Skipper will generate the code for you in that direction. And then once it’s done, you can run it as a service, either on your machine or anywhere you like.

Another example would be, let’s say you’re building a web app and you want, you know, some kind of back-end service that handles states, and handles users, and handles all sorts of things. Same idea. You write down what you want, and you let Skipper generate the code. And then whenever Skipper generates something, it also generates an openapi.json. And so this spec can then be consumed by other AI.

So for example, we have a really cool demo, that you’ll be able to see soon where we take, you know, a prompt that describes a relatively complex web app, right? That has many features and a lot of state and users interacting with each other, not just, you know, a static page or something that just works client side. And so we. Get Skipper to generate the front-end, and then we take the openapi.json and give it to a front-end generator, and we end up with a fully functioning website with multiple users and interactions in a couple of minutes. Pretty exciting stuff. But it can be used for pretty much anything as long — So it’s probabilistic, which means that there will be times where it doesn’t work, and it’s expected because it’s not perfect and AI cannot tackle everything. So especially if you go at frontier stuff, right? If you ask it to solve P=NP, write me a program that, you know, so that runs every NP program in polynomial time, right? How about you do that? That’s probably not gonna work. And, but if you stay within the scope of what, you know. If you stay within the scope of things that you can explain relatively simply, which is most business logic today, then it will be able to produce really good results.

[01:04:50] How Will the Software Developer Role Change in an AI-First World?

Henry Suryawirawan: Yeah, so we invite people to check it out. But one question, which kind of like in everyone’s mind at the moment. With that kinda like workflow, does it mean that the software developer role is kind of like changing by a lot? Because they won’t write the code anymore, they just write the specs. And what do you think the future is gonna be for software developer?

Julien Verlaguet: So first of all, I think software developers will keep on writing code, but the code that they will be writing is going to be much, much, much, much harder. It’s the kind of code that, you know, really matters, that every, you know, single character has to be carefully thought out, right? So this kind of code I think for the foreseeable future will be written by human beings. And I call it a blessing because we get to focus on the interesting stuff and all the boring stuff is taken care of by LLMs. So that’s great. And then the job, the the developer job is not gonna change all that much because it was never the job of developers to write code. Like writing code was just one small part of the job. It was never the job. The job was to architect, to design, to design testing structures, to design integration. It was always about design. It was never about writing code. And the code itself is not a detail. So what the code does and how it is structured is super important and developers will still do that. But I think the role of most developers is what resembled a tech lead before.

So before you started your career, and I would say there was three steps in your career, right? There was the, when you were a junior engineer, what you usually do is you help somebody who is more senior than you, right? And at this stage, people basically don’t trust you with what you’re producing. So they’ll give you things that if it doesn’t work, it’s okay because, and they’re going to come and work after you and make sure that everything that you did is actually correct, right? So in this phase, you’re learning and you’re not independent. Then there’s the phase where you become an independent engineer, which means that people think that they can give you pretty much anything. You will deliver it, it will be correct, and nobody will have to, you know, come after you and make sure that it’s correct. It doesn’t mean that they shouldn’t be code review, you know, they should be code review. But basically we are gonna trust your judgment when it comes to what you decide to do and how you decide to structure things, right? And then there is so, the tech lead, and so that’s senior engineers usually. And what tech leads do is they will drive efforts that are much larger and that involve other engineers, you know, doing parts of the coding.

And what has shifted now is that with LLMs, pretty much the job is to be a tech lead more than to be what used to be junior engineer. There are still, there’s still some work to do for people doing the coding, as I said. But most people who wanna stay in coding, they’ll have to evolve towards roles that are more like tech leads. And so they’ll have to have a bigger vision that involves multiple LLMs working at the same time. And how we going to integrate their work and how we gonna make sure that they don’t step in each other’s toes and how we going to make sense of all this work that’s happening. That’s, I think, where we headed.

Henry Suryawirawan: Wow, I like the way you, when you mentioned writing code is not the job itself. So yeah, thinking about it, I think it’s true, right? The designing thing, you know, the coming up with the testing, validation, the integration is always the hard problem. And also like interpreting the requirement in such a way that it can produce the correct, you know, like behavior and output, right? I think it’s, that’s also part of the design as well.

[01:09:06] 3 Tech Lead Wisdom

Henry Suryawirawan: So, Julien, it’s been a pleasure, pleasant conversation. You know, very excited to see this Skipper. But unfortunately we have to wrap up pretty soon. I only have one last question, which is the kind of like tradition in my podcast. I call this the three technical leadership wisdom. So think of it just like advice that you wanna give to the listeners. Maybe you can share your version today.

Julien Verlaguet: Right. So I think it’s going to be surprising because the ones, the things I care about are things that are not all that technical. So if I go back in my career, and you know, if I was able to talk to my 20-year-old self, this is what I would say. Number one, coding and programming and being a developer is a lot about relationships with other human beings, right? This whole thing that you’re gonna go in a cave and you’re gonna bang, write some code, and it’s gonna be awesome and everybody is gonna love it, this not true. This is not how things work, right? And so if I could talk to my 20-year-old self, I would say watch out for, you know, all those human aspects, right? Like when you are going to present something, like bear in mind that there are other people who want, who have sometimes different, you know, interests, sometimes different point of views. And this whole idea that may the best code win is, you know, juvenile. It’s not real. That’s not how things work, right? Like you, if you wanna grow in a role, in an engineering role, in a programming career, bear in mind that relationship with other humans are important, right? Number one.

Number two is take care about yourself and your health, you know. Like I worked very, very long hours and I remember one day I think I pulled an all-nighter. It was yet another time I did that in my twenties. And, you know, I, met this guy in the office and he told me, “You know, it’s not a sprint, it’s a marathon, you know. And so what you’re doing now. It is not sustainable.” And so I know that there are a lot of people who, you know, tell you that you have to work as hard as you can and as long as hours as you can. Run the experiments one time to not do that and see what happens. Because for me, what happened is in my twenties, I was in a relationship and my girlfriend back then did not want me to work past 6 or 7 PM something like that. Basically, if the computer was still on at 7:00 PM, you know, I was in a lot of trouble. And I realized that I was packing a lot more into my day because my window was now shorter and I realized how much more productive I was, you know. Like because I was sleeping well, I was sleeping much better. If you go to sleep at, you know, two in the morning, when you’re sleeping, you are actually debugging your code. That’s what you’re doing. You’re sleeping, you see the bugs in your mind and how are you gonna get good sleep this way, right?

And so the problems, and we all had that feeling right? We are in the zone, it’s working. We have 20 things running at the same time. We are thinking, if I go to bed now, I’m gonna forget all this context. And this is all bullshit, right? Go to bed, sleep. Go do something else, rest, and you will see. My advice is for people, is try it for one month and actually measure the output and tell me I’m wrong after that. Now, we all understand that if you have a deadline, if you have something, you know, that’s exceptional, right? Like of course you are late on something, it’s something that you really care about. We all understand that sometimes you’re gonna work longer hours, but it’s not sustainable. So what’s going to happen is every time you do that, you’ll pay the price for it later. And so then you’ll be less productive for the next two weeks. If you’re happy with that, then maybe it’s the right time to do that. If you’re not, then don’t do that.

So first wisdom was relationships. Second one is take care of yourself. Third one is be curious. And try new technologies, try new things. Like in my career, the people who I know who have had the best careers and who have been the best developers to work with, they knew stuff. They tried, so for example, they tried programming languages nobody else has heard of, right? They spent a weekend actually trying it, right, and seeing how it feels. Now it’s true that, as I said earlier, we have an emotional relationship with our tools and with our programming languages, and that’s because we’ve built awesome things with those tools. And usually we remember all these awesome things that we’ve built. And so it’s very hard for us to, it’s like I guess if you are into surfing, right? Like I’m pretty sure you have a relationship with your board that is like emotional. Like you’ve done great stuff with this board. And if somebody comes along and says, well, maybe you should try this other side, oh fuck that. I have my great board that I love and nobody else, when nothing else is worth. So I would say, try out different things. Go try out a programming language you’ve never tried before. Go try out a different database. Go try out.

So for example, a divide that you see very often which I think is a pity, is a divide between front-end and back-end engineers, right? And a lot of back-end engineers don’t even wanna try to see what it is to do front-end work or vice versa. And how are you going to have empathy in an organization when you don’t understand where people are coming from, right? So even if you are a hardcore, you know, Linux kernel guy who’s written, you know, stuff in a kernel all day long, you should go learn how to do some CSS. You should learn how to do that. Because you should have empathy with people who spend all day trying to get those pixels to align. And if you do, you’ll be a better engineer. When you will be part of a bigger organization, there will be people with those kind of jobs. And the last thing they need is somebody who believes that, you know, it’s all about coding and all about the stuff that I’m coding because everybody else is an idiot. That’s the last thing an organization needs. So these are my three advice. I guess they’re not, maybe not as technical as you were expecting, but.

Henry Suryawirawan: Yeah, I mean, that’s actually perfect, right? I really love all of them, right? Relationship, take care of yourself. And the last thing, be curious so that you can have empathy, you know, for the other relationship with the other humans, right? I think that’s also important.

So Julien, if people would love to connect with you, ask you more questions, is there a place where they can find you online?

Julien Verlaguet: So I have… Yeah, we have a Discord. We have a Discord for Skip — So we are going to create a Discord for Skipper, which we haven’t created yet. But we have a Discord for Skip. And so you can find me there. I’m always connected. And, what else? No, that’s it. I’m on Discord.

Henry Suryawirawan: Cool. I’ll put it in the show notes when it’s ready. So again, thank you so much for sharing. I’m really excited to see how Skipper will change the game, you know, on how, you know, AI agents, LLM would perform much better in terms of producing, you know, better pieces of code, right? So better services and so that humans can be much more productive. So yeah, hope you have a good luck, you know, with your project and hope to see the impact that you make in the software engineering world.

Julien Verlaguet: Thanks a lot for having me, Henry. This was a lot of fun. Anytime. And yes, I hope we get to do this another time, maybe.

– End –