Lalit Maganti (Tag: Ai)

AI agents are not subsystem maintainers

Mon, 27 Jul 2026 10:43:00 +0100

Antirez, the creator of Redis, recently argued that expert programmers using AI should think of themselves as being Linus Torvalds.

The key part of his analogy is this:

Automatic programming, instead, in the hands of people that are expert technicians, or expert programmers, expert designers, expert software architects, is to assume the role of Linus, with the agents and the LLMs assuming the role of the different maintainers of the different subsystems.

This analogy does not work for me. As they exist today, I cannot be Linus and an LLM cannot be my subsystem maintainer.

Why? In one word: trust. Linus trusts his subsystem maintainers. He is able to focus on the big picture, safe in the knowledge that his lieutenants will consistently exercise good judgment in the long-term maintenance of their subsystems. He knows this because they have earned his trust by proving themselves over and over again.

Every time I have tried to give current LLMs the opportunity to prove themselves in this way, I’ve ended up regretting it. At a high level, I simply dislike too many of the decisions they make. This has happened even in areas (performance, databases, developer tool UX) where I feel qualified to judge how well an AI is doing. I’ve written about this in depth in my account of building Syntaqlite with AI, but beyond this, I’ve used LLMs in many different ways both in my job and in other side projects.

If a human behaved like this repeatedly, I would not give them ownership of any part of a project I own. Trust has to be earned, and current agents have not come close to earning enough of it for me to step back. It is much cheaper for me to stay involved while the design is taking shape than to discover at the end that a sequence of plausible local decisions added up to the wrong system.

And no, just passing tests or static verification is not enough for me. The decisions I most often dislike are precisely the ones that cannot be easily checked. Tests cannot tell me that an agent chose the wrong abstraction or produced an API that is unpleasant to use. They cannot tell me that the code will be difficult to change in three years, and they certainly cannot tell me that I have solved the wrong problem.

None of this makes AI a bad tool. With clear parameters and enough steering, it is an extremely capable bounded implementer. AI has substantially changed what I can build, and I use it extensively every day.

But calling it a maintainer is a step too far. If I cannot trust it to exercise independent judgment, then I still have to own the code. At best, today’s AI is helping me implement my decisions; it is not freeing me to be Linus.

Perfetto v57: fixing PyTorch traces, plus journald logs and an AI skill

Fri, 03 Jul 2026 00:23:00 +0100

We just released Perfetto v57 and I wanted to share the new things I’m most excited about. This is something I wanted to do for past releases but I just never quite got round to it. It’s also something I plan on doing more of going forward: there might even be dedicated pieces if I think the feature deserves it!

PyTorch traces reliably work!#

What I’m most excited about in this release isn’t a feature but a bugfix. If you used the PyTorch profiler and opened the resulting trace in Perfetto, there was a decent chance some of your events would just not show up; specifically this would happen when these events overlapped each other on a single track.

Technically, PyTorch is in the wrong here. The Chrome Trace Event (JSON) format says duration events on a track have to nest and can’t overlap; if you need overlap, you’re supposed to use async events. chrome://tracing appears to handle them, but its rendering is actually buggy as soon as a trace has real overlaps; people just learned to live with it. So when a bug came in January about overlapping events being broken, I closed it as working-as-intended because I couldn’t see any easy fix from our end.

Then over time, I realized it wasn’t just one person with a weird trace. The bug had affected many different PyTorch traces, and even JAX traces. I even discovered that people were maintaining forks and hosting old versions of Perfetto just to deal with this issue. This is basically Hyrum’s Law: it doesn’t much matter what the spec says, once enough people rely on some behaviour, that behaviour is your contract.

This is also a very similar situation to what I discussed in Perfetto and open source: it was genuinely hard for me to tell just how many people it was affecting at the beginning: if I had understood, I would have dealt with this very differently.

In any case, in v57, Perfetto now imports the overlapping events instead of dropping them, overflowing the extras onto a different line in the same UI track. I do have to credit @i_komarov who helped me understand how big this issue was and to converge on the fix.

Recording systemd journald logs#

Perfetto has always tried to give you as much context as possible about what the system is doing and put it all on a single timeline. One missing piece of that on Linux has been in logs from journald; there’s a ton of useful information there but until now it was hard to correlate that with e.g. scheduling or other app level information.

With v57, there’s now a linux.systemd_journald data source that records journald entries straight into the trace, on the same timeline as everything else. You can filter by minimum syslog priority, by SYSLOG_IDENTIFIER, or by systemd unit, so you are not pulling in the entire journal when you only care about one service.

To show it in action, here’s a trace I recorded on my own machine. Alongside a few log lines I generated myself, it happened to capture a real udisks SMART error that fired while I was recording; exactly the kind of thing you’d otherwise miss.

I have to shout out @staticfloat who contributed this feature; thanks a lot for doing this!

An AI skill for recording and analysis#

For a while now, people have been building their own AI tooling around Perfetto. SmartPerfetto and perfetto-mcp are two public examples I’ve come across but many teams inside Google have built their versions of these which each handle a subset of Perfetto’s functionality. To us, this is a sign that there’s demand out there to build something upstream so that everyone can benefit.

v57 includes an installable AI skill that works with Claude Code, Codex, Pi, Opencode, Antigravity, and anything else that speaks the Agent Skills format. It explains to the agent how to record traces (Android only for now!) and then query them. It also includes dedicated workflows for debugging Android memory, a specific area of focus for us; we plan on expanding these workflows over time including to Linux.

The skill is just one piece of a much larger AI plan, which I wrote up as an RFC. We will be doing much more in this space in the coming months.

State tracks#

State tracks are a new kind of primitive in Perfetto, sitting alongside slices and counters. They represent something that is always in exactly one state: a connection that is open or closed, a device that is on or off, a worker that is idle or busy. You could already model this with slices, by emitting a slice that starts when a state begins and ends when it changes. This works but you have to be careful to emit events exactly at the same timestamps or you end up having gaps.

With a state track you don’t emit start and end events at all. At each point where the state changes, you emit one event that says what the state is now. The previous state ends automatically. Clearing the track marks it idle. This just makes it a lot simpler to manage.

While the simpler emission model is the obvious win, the reason we made it a first-class primitive goes further than that. Because a state track holds exactly one value at any moment, it can never overlap itself, and the trace processor can lean on that guarantee to treat it specially. You can attach arguments that change as the state does. And in ring-buffer traces, where old events get overwritten as the buffer wraps, you can periodically re-emit the current state, so a trace that starts mid-stream still knows what state everything is in.

State tracks are still quite new so there are likely to be some rough edges here, but we think they’ll become an important primitive: because they’re guaranteed never to overlap, you can do things with them you simply couldn’t before.

Querying traces over HTTPS#

I recently discovered that quite a few folks have written scripts which download a trace and then run queries against it with the trace_processor CLI. Which made me think: why don’t we just support that natively inside the CLI itself?

So in v57, trace_processor takes an http(s) URL or a ui.perfetto.dev share link directly. It streams the trace and runs your query with no local copy. Here it is against the journald trace from the previous section:

$ trace_processor query \
 "https://lalitm.com/traces/journald-demo.pftrace" \
 "select log_source, count(*) as n from logs where msg is not null group by log_source"

"log_source","n"
"systemd_journald",30

It’s one of those small polish features which you wouldn’t necessarily think of immediately but when you have it, you’d never want to go back.

Lightning round#

A few smaller things also landed:

Shift + mouse wheel scrolls the timeline horizontally. A tiny change that removes a daily papercut when you are panning around a wide trace.
The query results grid got a real overhaul. You can sort, reorder, and hide columns, the sidebar collapses, and SQL errors now show up in the grid instead of disappearing.
Adjacent slices that share a boundary render with a 1px gap, so a run of back-to-back slices reads as separate slices rather than one solid block.
Saved textproto record configs are editable, saving the trouble of downloading the config, deleting it, editing it locally, and then reuploading.

Wrap up#

Overall this was a pretty solid release of Perfetto; it’s always satisfying to fix a bug blocking a lot of people. AI skills and state tracks are both first steps in a much longer journey, and journald logs are an important new capability that improves our Linux support. The rest are mostly a bunch of small quality of life improvements which are always nice to see!

Security analysis is finally reaching software's long tail

Sun, 07 Jun 2026 16:26:00 +0100

This post was formerly titled “17 bugs in 10 weeks from AI security scanning”. It was changed upon feedback from trusted readers who suggested, while accurate, it did not give a good sense of what I’m actually arguing in the post.

Over the last several weeks, I’ve been receiving more security bug reports for Perfetto’s trace processor than I ever have before, all of them found by AI. And I’m very happy about it! These are bugs that would almost certainly not have been found a year ago and it feels good to close these loopholes even though trace processor is by no means security critical.

For years, security researchers concentrated their time on the highest-stakes targets: kernels, cryptography libraries, password managers. But there’s a lot of code out there which is security-relevant but not truly security-critical. In my experience, these sorts of projects didn’t draw much attention. Now systems in the long tail can get that attention which they wouldn’t have before.

Why is this happening#

Trace processor is a project which sits squarely in that long tail. It’s a C++ library (yes, Rust would be the obvious choice today but it’s not practical to rewrite, see footnote ¹) for processing recorded traces of various formats. These are typically traces you collected yourself or in your test infra and process offline so “untrusted input” isn’t much of a concern.

However, some people do process traces they didn’t collect themselves (e.g. user bug reports, automated collection from dogfood users). For those cases we’ve strongly recommended sandboxing trace processor (e.g. gvisor, sandbox2, or minijail) or, for even more sensitive use cases, a VM.

Beyond sandboxing, for catching issues proactively, we mainly relied on fuzzing running internally in Google. These fuzzers occasionally surfaced real, actionable bugs: we set them up to pass in arbitrary trace bytes (as this is the main “attack surface”) but over time these became quite rare as they discovered much of the low hanging fruit, which we quickly fixed. The bugs that remained tend to live deep in the internals, reachable only with a very precisely crafted sequence of bytes that a fuzzer is unlikely to hit by mutation alone.

Apart from this, there has rarely been any bandwidth or resources for a human, either a security expert or someone from my team, to spend lots of time finding security issues in trace processor. There were always other parts of Perfetto more worth spending security time on (e.g. the tracing service, on-device profilers) as they’re actively running in production systems.

All of this changed as of a couple of months ago. We started receiving bug reports filed by some central team which appears to be running AI-based security scanning against various projects throughout Google. Unfortunately, I have to be hand wavy about what exactly they’re doing as their work doesn’t appear to be public.

Starting in early April, we had a slow drip of 1 bug a week, but since the end of April this increased to a rate of several a week, with some days having 3 or 4 being opened in quick succession. This lasted until mid-May, at which point it started tapering back to 1-2 a week with some weeks having none.

I also want to say that the quality of the bugs is high. They’re well-described, often with the relevant attacker model already worked out and even minimal fixes proposed: basically everything I could ask for from a bug report. This matches what both curl and Linux kernel maintainers have noted about security bugs they’ve received, especially how sharply quality has improved in the last few months.

As I can only see the bugs that get filed against me, not the raw output of the AI scanner, I don’t know exactly how much triage happens upstream. My guess is there’s a human doing a light pass to drop obvious noise before reports reach client teams, but judging from the rate at which bugs are opened and the way they’re filed, I doubt anyone is deeply triaging each one.

In total, we’ve received 21 bugs (17 real issues and 4 not actionable), which can be broken down into the following categories:

10 bounds checking: arbitrary trace data flowing into fixed-size buffers or unchecked array indices, leading to out-of-bounds reads or writes.
5 use-after-free: back-pointers, pointer snapshots, or hashmap keys outliving the object they refer to.
1 stack overflow: unbounded recursion when input is deeply nested.
1 access control: not enforcing allowlists on some rare codepaths.
4 closed as not actionable: either where the chance of exploit was purely hypothetical or where fixing would have required fundamental design changes which were not worth the tiny security risk.

All 17 real issues have been fixed, almost all shipping in Perfetto v56.0 ².

How it feels to get a report#

How does receiving one of these reports actually feel? Well not as bad as you’d think. Unlike a security-critical application like OpenSSL or curl, in trace processor, a security issue is very unlikely to be a P0 I have to drop everything to fix. Don’t get me wrong, it’s still a priority but one where I have the luxury of taking a few days to figure out the right answer and can release fixes according to our normal schedule, instead of trying to rush out a CVE and get everyone to patch immediately.

Also thankfully, because the majority of the issues are mechanical, the fixes are generally quite straightforward. Take this PR, for example:

We build a key string into a fixed-size stack buffer while parsing some metadata.
The bounds check only runs in debug builds, and the metadata name comes straight from the trace. Putting a long enough name means you would escape the buffer.
The fix is a simple matter of swapping the stack buffer for a std::string. The code path is very cold (only once or twice in a trace) so the extra heap allocation doesn’t matter.

In fact, these sorts of issues are so mechanical that I trust a coding agent to just fix them with minimal guidance: take the well-written report, feed it to the agent, and within ~10 minutes there’s a 10-20 line PR which fixes it. I review every line thoroughly and make sure I understand it, but these tasks are not difficult and firmly inside the “jagged frontier” of what AI can do.

I want to stress though that not every issue is mechanical or can be left to AI; a few reports actually point more to design problems than incorrect function implementations. This use after free is a good example:

The problem was a state object held a back-pointer that could end up pointing to freed memory given certain data appearing in the trace.
The immediate dangling case was easy to patch by just having a callback which invalidated the back-pointer on free. But this is a horrible hack which makes the lifetimes of the objects involved impossible to reason about.
The real problem here is that you had a child object whose parent could go away before it, which really shouldn’t happen if this code is properly architected.
Fixing it properly meant restructuring the ownership model so the lifetime was correct by construction.

The interesting thing was that this was a problem I was aware of and that I had been meaning to clean up for close to a year but never got round to: the security bug just gave me the push and justification to do it. This applied in a couple of other bugs as well and made me internalize that security issues can sometimes be correlated with deeper design flaws or hacky code so there are wider benefits to “security scanning” than just the direct bugs they find.

Will this last?#

One thing I am wary of is how long this stream of bugs will keep up; I’m feeling good about it given it’s only been going on for a couple of months, but I can easily imagine that if this goes on for several more months, it might become mentally exhausting.

But my suspicion is that this will go to zero. Why? It’s to do with the pattern of how these bugs are being filed. Each part of the codebase seems like it’s getting a day or two of attention (and associated bugs) before moving on to a different part. Repeats are rare, and the pace of bugs has slowed especially in the last couple of weeks: we had a lot more in the start of May (several a week) but now we’re down to 1-2 a week. There are a finite number of files, so eventually my gut tells me they will run out.

An important consideration is whether we’ll add new bugs faster than the scanner can find old ones. My suspicion is no; the 17 real issues so far are from scanning across 9 years of development. Even if that number triples before things settle, the scanner is still working through years of accumulated code. And we wrote a lot more code, a lot faster, in the earlier years of the project, so the rate of new code being added now is lower than it once was.

The other question is whether new model releases will find more complex design issues rather than the simple issues we’re finding today. Those take significantly more time and effort to fix and so would be a lot more painful if we were to get many of those. I’m very unsure on this so we’ll just have to wait and see!

Where this leaves us#

I feel Daniel Stenberg (curl maintainer) phrased it well in this post:

Any project that has not scanned their source code with AI powered tooling will likely find huge number of flaws, bugs and possible vulnerabilities with this new generation of tools.

This rings very true to me. More broadly, I think folks will have one of three experiences:

Untrusted input + security critical (e.g. curl, kernel, OpenSSL): many complex reports, with a higher false positive rate than the other categories, because there’s a lot of attention on the project and much of the low hanging fruit would already have been picked in the critical codepaths. Though codepaths for lesser-used functionality (e.g. legacy drivers) could end up in category 2 instead.
Untrusted input + not previously audited (e.g. trace processor): a wave of mechanical bugs at a manageable pace and low individual stress because the project is not on a security critical code path. This is where both Daniel and I expect AI security scanners to have the most impact.
No untrusted input (internal tools, math libs, anything operating only on trusted data): you probably won’t notice this shift at all.

My own case sits squarely in that second bucket. But I don’t want to over-generalize from my experience, because there are three things that make this manageable for me that wouldn’t be true for everyone: a) I’m paid to maintain trace processor as part of my full time job; b) someone else is taking the effort to run the AI scans and discover the bugs in the first place; c) the reports appear to be lightly filtered by an upstream human reviewer, enough to strip obvious noise but probably not a deep triage.

To me, this points to a gap in the ecosystem: most open-source projects cannot afford to have a dedicated team doing security scanning for them, and telling a maintainer to stand up their own pipeline when their security risk is marginal will restrict this to only the most motivated projects. I would guess we’re going to see a lot more innovation in this space, including from the big AI labs.

All in all, I’m cautiously positive about my own experience: most of the bugs are mechanical, a few have nudged long-overdue design cleanups, and the pace is manageable. There’s plenty I don’t know about how this evolves: whether the pace holds, whether future models start finding harder design issues. So this should very much be treated as a snapshot, not a forecast!

A common response I expect is: “if it’s parsing arbitrary binary data like traces, it should be written in Rust.” In a vacuum I agree and if I was writing trace processor from scratch today, I would definitely use Rust. But switching to Rust is unfortunately quite impractical; the library is a significant amount of code and is embedded in hundreds of downstream tools, many in environments that don’t have a Rust toolchain. Asking all our embedders to start using Rust would be a significant burden and one I don’t want to impose. Not to mention that our team doesn’t actually have any Rust expertise so reviewing this code at a standard I want from trace processor would be a significant productivity hit until folks got up to speed. And unlike others in the industry, I don’t feel comfortable just rewriting the whole project in one shot and calling it a day… ↩︎
The couple of remaining bugs were found after the v56.0 release was cut and are low-priority enough that it’s not worth rushing out a release for them. They have already been fixed on main and will be fixed in an upcoming point release. ↩︎

Eight years of wanting, three months of building with AI

Sun, 05 Apr 2026 13:00:00 +0100

For eight years, I’ve wanted a high-quality set of devtools for working with SQLite. Given how important SQLite is to the industry¹, I’ve long been puzzled that no one has invested in building a really good developer experience for it².

A couple of weeks ago, after ~250 hours of effort over three months³ on evenings, weekends, and vacation days, I finally released syntaqlite (GitHub), fulfilling this long-held wish. And I believe the main reason this happened was because of AI coding agents⁴.

Of course, there’s no shortage of posts claiming that AI one-shot their project or pushing back and declaring that AI is all slop. I’m going to take a very different approach and, instead, systematically break down my experience building syntaqlite with AI, both where it helped and where it was detrimental.

I’ll do this while contextualizing the project and my background so you can independently assess how generalizable this experience was. And whenever I make a claim, I’ll try to back it up with evidence from my project journal, coding transcripts, or commit history⁵.

Why I wanted it#

In my work on Perfetto, I maintain a SQLite-based language for querying performance traces called PerfettoSQL. It’s basically the same as SQLite but with a few extensions to make the trace querying experience better. There are ~100K lines of PerfettoSQL internally in Google and it’s used by a wide range of teams.

Having a language which gets traction means your users also start expecting things like formatters, linters, and editor extensions. I’d hoped that we could adapt some SQLite tools from open source but the more I looked into it, the more disappointed I was. What I found either wasn’t reliable enough, fast enough⁶, or flexible enough to adapt to PerfettoSQL. There was clearly an opportunity to build something from scratch, but it was never the “most important thing we could work on”. We’ve been reluctantly making do with the tools out there but always wishing for better.

On the other hand, there was the option to do something in my spare time. I had built lots of open source projects in my teens⁷ but this had faded away during university when I felt that I just didn’t have the motivation anymore. Being a maintainer is much more than just “throwing the code out there” and seeing what happens. It’s triaging bugs, investigating crashes, writing documentation, building a community, and, most importantly, having a direction for the project.

But the itch of open source (specifically freedom to work on what I wanted while helping others) had never gone away. The SQLite devtools project was eternally in my mind as “something I’d like to work on”. But there was another reason why I kept putting it off: it sits at the intersection of being both hard and tedious.

What makes it hard and tedious#

If I was going to invest my personal time working on this project, I didn’t want to build something that only helped Perfetto: I wanted to make it work for any SQLite user out there⁸. And this means parsing SQL exactly like SQLite.

The heart of any language-oriented devtool is the parser. This is responsible for turning the source code into a “parse tree” which acts as the central data structure anything else is built on top of. If your parser isn’t accurate, then your formatters and linters will inevitably inherit those inaccuracies; many of the tools I found suffered from having parsers which approximated the SQLite language rather than representing it precisely.

Unfortunately, unlike many other languages, SQLite has no formal specification describing how it should be parsed. It doesn’t expose a stable API for its parser either. In fact, quite uniquely, in its implementation it doesn’t even build a parse tree at all⁹! The only reasonable approach left in my opinion is to carefully extract the relevant parts of SQLite’s source code and adapt it to build the parser I wanted¹⁰.

This means getting into the weeds of SQLite source code, a fiendishly difficult codebase to understand. The whole project is written in C in an incredibly dense style; I’ve spent days just understanding the virtual table API¹¹ and implementation. Trying to grasp the full parser stack was daunting.

There’s also the fact that there are >400 rules in SQLite which capture the full surface area of its language. I’d have to specify in each of these “grammar rules” how that part of the syntax maps to the matching node in the parse tree. It’s extremely repetitive work; each rule is similar to all the ones around it but also, by definition, different.

And it’s not just the rules but also coming up with and writing tests to make sure it’s correct, debugging if something is wrong, triaging and fixing the inevitable bugs people filed when I got something wrong…

For years, this was where the idea died. Too hard for a side project¹², too tedious to sustain motivation, too risky to invest months into something that might not work.

How it happened#

I’ve been using coding agents since early 2025 (Aider, Roo Code, then Claude Code since July) and they’d definitely been useful but never something I felt I could trust a serious project to. But towards the end of 2025, the models seemed to make a significant step forward in quality¹³. At the same time, I kept hitting problems in Perfetto which would have been trivially solved by having a reliable parser. Each workaround left the same thought in the back of my mind: maybe it’s finally time to build it for real.

I got some space to think and reflect over Christmas and decided to really stress test the most maximalist version of AI: could I vibe-code the whole thing using just Claude Code on the Max plan (£200/month)?

Through most of January, I iterated, acting as semi-technical manager and delegating almost all the design and all the implementation to Claude. Functionally, I ended up in a reasonable place: a parser in C extracted from SQLite sources using a bunch of Python scripts, a formatter built on top, support for both the SQLite language and the PerfettoSQL extensions, all exposed in a web playground.

But when I reviewed the codebase in detail in late January, the downside was obvious: the codebase was complete spaghetti¹⁴. I didn’t understand large parts of the Python source extraction pipeline, functions were scattered in random files without a clear shape, and a few files had grown to several thousand lines. It was extremely fragile; it solved the immediate problem but it was never going to cope with my larger vision, never mind integrating it into the Perfetto tools. The saving grace was that it had proved the approach was viable and generated more than 500 tests, many of which I felt I could reuse.

I decided to throw away everything and start from scratch while also switching most of the codebase to Rust¹⁵. I could see that C was going to make it difficult to build the higher level components like the validator and the language server implementation. And as a bonus, it would also let me use the same language for both the extraction and runtime instead of splitting it across C and Python.

More importantly, I completely changed my role in the project. I took ownership of all decisions¹⁶ and used it more as “autocomplete on steroids” inside a much tighter process: opinionated design upfront, reviewing every change thoroughly, fixing problems eagerly as I spotted them, and investing in scaffolding (like linting, validation, and non-trivial testing¹⁷) to check AI output automatically.

The core features came together through February and the final stretch (upstream test validation, editor extensions, packaging, docs) led to a 0.1 launch in mid-March.

But in my opinion, this timeline is the least interesting part of this story. What I really want to talk about is what wouldn’t have happened without AI and also the toll it took on me as I used it.

AI is why this project exists, and why it’s as complete as it is#

Overcoming inertia#

I’ve written in the past about how one of my biggest weaknesses as a software engineer is my tendency to procrastinate when facing a big new project. Though I didn’t realize it at the time, it could not have applied more perfectly to building syntaqlite.

AI basically let me put aside all my doubts on technical calls, my uncertainty of building the right thing and my reluctance to get started by giving me very concrete problems to work on. Instead of “I need to understand how SQLite’s parsing works”, it was “I need to get AI to suggest an approach for me so I can tear it up and build something better"¹⁸. I work so much better with concrete prototypes to play with and code to look at than endlessly thinking about designs in my head, and AI lets me get to that point at a pace I could not have dreamed about before. Once I took the first step, every step after that was so much easier.

Faster at churning code#

AI turned out to be better than me at the act of writing code itself, assuming that code is obvious. If I can break a problem down to “write a function with this behaviour and parameters” or “write a class matching this interface,” AI will build it faster than I would and, crucially, in a style that might well be more intuitive to a future reader. It documents things I’d skip, lays out code consistently with the rest of the project, and sticks to what you might call the “standard dialect” of whatever language you’re working in¹⁹.

That standardness is a double-edged sword. For the vast majority of code in any project, standard is exactly what you want: predictable, readable, unsurprising. But every project has pieces that are its edge, the parts where the value comes from doing something non-obvious. For syntaqlite, that was the extraction pipeline and the parser architecture. AI’s instinct to normalize was actively harmful there, and those were the parts I had to design in depth and often resorted to just writing myself.

But here’s the flip side: the same speed that makes AI great at obvious code also makes it great at refactoring. If you’re using AI to generate code at industrial scale, you have to refactor constantly and continuously²⁰. If you don’t, things immediately get out of hand. This was the central lesson of the vibe-coding month: I didn’t refactor enough, the codebase became something I couldn’t reason about, and I had to throw it all away. In the rewrite, refactoring became the core of my workflow. After every large batch of generated code, I’d step back and ask “is this ugly?” Sometimes AI could clean it up. Other times there was a large-scale abstraction that AI couldn’t see but I could; I’d give it the direction and let it execute²¹. If you have taste, the cost of a wrong approach drops dramatically because you can restructure quickly²².

Teaching assistant#

Of all the ways I used AI, research had by far the highest ratio of value delivered to time spent.

I’ve worked with interpreters and parsers before but I had never heard of Wadler-Lindig pretty printing²³. When I needed to build the formatter, AI gave me a concrete and actionable lesson from a point of view I could understand and pointed me to the papers to learn more. I could have found this myself eventually, but AI compressed what might have been a day or two of reading into a focused conversation where I could ask “but why does this work?” until I actually got it.

This extended to entire domains I’d never worked in. I have deep C++ and Android performance expertise but had barely touched Rust tooling or editor extension APIs. With AI, it wasn’t a problem: the fundamentals are the same, the terminology is similar, and AI bridges the gap²⁴. The VS Code extension would have taken me a day or two of learning the API before I could even start. With AI, I had a working extension within an hour.

It was also invaluable for reacquainting myself with parts of the project I hadn’t looked at for a few days²⁵. I could control how deep to go: “tell me about this component” for a surface-level refresher, “give me a detailed linear walkthrough” for a deeper dive, “audit unsafe usages in this repo” to go hunting for problems. When you’re context switching a lot, you lose context fast. AI let me reacquire it on demand.

More than I’d have built alone#

Beyond making the project exist at all, AI is also the reason it shipped as complete as it did. Every open source project has a long tail of features that are important but not critical: the things you know theoretically how to do but keep deprioritizing because the core work is more pressing. For syntaqlite, that list was long: editor extensions, Python bindings, a WASM playground, a docs site, packaging for multiple ecosystems²⁶. AI made these cheap enough that skipping them felt like the wrong trade-off.

It also freed up mental energy for UX²⁷. Instead of spending all my time on implementation, I could think about what a user’s first experience should feel like: what error messages would actually help them fix their SQL, how the formatter output should look by default, whether the CLI flags were intuitive. These are the things that separate a tool people try once from one they keep using, and AI gave me the headroom to care about them. Without AI, I would have built something much smaller, probably no editor extensions or docs site. AI didn’t just make the same project faster. It changed what the project was.

Where AI had its costs#

The addiction#

There’s an uncomfortable parallel between using AI coding tools and playing slot machines²⁸. You send a prompt, wait, and either get something great or something useless. I found myself up late at night wanting to do “just one more prompt,” constantly trying AI just to see what would happen even when I knew it probably wouldn’t work. The sunk cost fallacy kicked in too: I’d keep at it even in tasks it was clearly ill-suited for, telling myself “maybe if I phrase it differently this time.”

The tiredness feedback loop made it worse²⁹. When I had energy, I could write precise, well-scoped prompts and be genuinely productive. But when I was tired, my prompts became vague, the output got worse, and I’d try again, getting more tired in the process. In these cases, AI was probably slower than just implementing something myself, but it was too hard to break out of the loop³⁰.

Losing touch#

Several times during the project, I lost my mental model of the codebase³¹. Not the overall architecture or how things fitted together. But the day-to-day details of what lived where, which functions called which, the small decisions that accumulate into a working system. When that happened, surprising issues would appear and I’d find myself at a total loss to understand what was going wrong. I hated that feeling.

The deeper problem was that losing touch created a communication breakdown³². When you don’t have the mental thread of what’s going on, it becomes impossible to communicate meaningfully with the agent. Every exchange gets longer and more verbose. Instead of “change FooClass to do X,” you end up saying “change the thing which does Bar to do X”. Then the agent has to figure out what Bar is, how that maps to FooClass, and sometimes it gets it wrong³³. It’s exactly the same complaint engineers have always had about managers who don’t understand the code asking for fanciful or impossible things. Except now you’ve become that manager.

The fix was deliberate: I made it a habit to read through the code immediately after it was implemented and actively engage to see “how would I have done this differently?”.

Of course, in some sense all of the above is also true of code I wrote a few months ago (hence the sentiment that AI code is legacy code), but AI makes the drift happen faster because you’re not building the same muscle memory that comes from originally typing it out.

The slow corrosion#

There were some other problems I only discovered incrementally over the three months.

I found that AI made me procrastinate on key design decisions³⁴. Because refactoring was cheap, I could always say “I’ll deal with this later.” And because AI could refactor at the same industrial scale it generated code, the cost of deferring felt low. But it wasn’t: deferring decisions corroded my ability to think clearly because the codebase stayed confusing in the meantime. The vibe-coding month was the most extreme version of this. Yes, I understood the problem, but if I had been more disciplined about making hard design calls earlier, I could have converged on the right architecture much faster.

Tests created a similar false comfort³⁵. Having 500+ tests felt reassuring, and AI made it easy to generate more. But neither humans nor AI are creative enough to foresee every edge case you’ll hit in the future; there are several times in the vibe-coding phase where I’d come up with a test case and realise the design of some component was completely wrong and needed to be totally reworked. This was a significant contributor to my lack of trust and the decision to scrap everything and start from scratch.

Basically, I learned that the “normal rules” of software still apply in the AI age: if you don’t have a fundamental foundation (clear architecture, well-defined boundaries) you’ll be left eternally chasing bugs as they appear.

No sense of time#

Something I kept coming back to was how little AI understood about the passage of time³⁶. It sees a codebase in a certain state but doesn’t feel time the way humans do. I can tell you what it feels like to use an API, how it evolved over months or years, why certain decisions were made and later reversed.

The natural problem from this lack of understanding is that you either make the same mistakes you made in the past and have to relearn the lessons or you fall into new traps which were successfully avoided the first time, slowing you down in the long run. In my opinion, this is a similar problem to why losing a high-quality senior engineer hurts a team so much: they carry history and context that doesn’t exist anywhere else and act as a guide for others around them.

In theory, you can try to preserve this context by keeping specs and docs up to date. But there’s a reason we didn’t do this before AI: capturing implicit design decisions exhaustively is incredibly expensive and time-consuming to write down. AI can help draft these docs, but because there’s no way to automatically verify that it accurately captured what matters, a human still has to manually audit the result. And that’s still time-consuming.

There’s also the context pollution problem. You never know when a design note about API A will echo in API B. Consistency is a huge part of what makes codebases work, and for that you don’t just need context about what you’re working on right now but also about other things which were designed in a similar way. Deciding what’s relevant requires exactly the kind of judgement that institutional knowledge provides in the first place.

Relativity#

Reflecting on the above, the pattern of when AI helped and when it hurt was fairly consistent.

When I was working on something I already understood deeply, AI was excellent. I could review its output instantly, catch mistakes before they landed and move at a pace I’d never have managed alone. The parser rule generation is the clearest example³⁷: I knew exactly what each rule should produce, so I could review AI’s output within a minute or two and iterate fast.

When I was working on something I could describe but didn’t yet know, AI was good but required more care. Learning Wadler-Lindig for the formatter was like this: I could articulate what I wanted, evaluate whether the output was heading in the right direction, and learn from what AI explained. But I had to stay engaged and couldn’t just accept what it gave me.

When I was working on something where I didn’t even know what I wanted, AI was somewhere between unhelpful and harmful. The architecture of the project was the clearest case: I spent weeks in the early days following AI down dead ends, exploring designs that felt productive in the moment but collapsed under scrutiny. In hindsight, I have to wonder if it would have been faster just thinking it through without AI in the loop at all.

But expertise alone isn’t enough. Even when I understood a problem deeply, AI still struggled if the task had no objectively checkable answer³⁸. Implementation has a right answer, at least at a local level: the code compiles, the tests pass, the output matches what you asked for. Design doesn’t. We’re still arguing about OOP decades after it first took off.

Concretely, I found that designing the public API of syntaqlite was where this hit home the hardest. I spent several days in early March doing nothing but API refactoring, manually fixing things any experienced engineer would have instinctively avoided but AI made a total mess of. There’s no test or objective metric for “is this API pleasant to use” and “will this API help users solve the problems they have” and that’s exactly why the coding agents did so badly at it.

This takes me back to the days I was obsessed with physics and, specifically, relativity. The laws of physics look simple and Newtonian in any small local area, but zoom out and spacetime curves in ways you can’t predict from the local picture alone. Code is the same: at the level of a function or a class, there’s usually a clear right answer, and AI is excellent there. But architecture is what happens when all those local pieces interact, and you can’t get good global behaviour by stitching together locally correct components.

Knowing where you are on these axes at any given moment is, I think, the core skill of working with AI effectively.

Wrap-up#

Eight years is a long time to carry a project in your head. Seeing these SQLite tools actually exist and function after only three months of work is a massive win, and I’m fully aware they wouldn’t be here without AI.

But the process wasn’t the clean, linear success story people usually post. I lost an entire month to vibe-coding. I fell into the trap of managing a codebase I didn’t actually understand, and I paid for that with a total rewrite.

The takeaway for me is simple: AI is an incredible force multiplier for implementation, but it’s a dangerous substitute for design. It’s brilliant at giving you the right answer to a specific technical question, but it has no sense of history, taste, or how a human will actually feel using your API. If you rely on it for the “soul” of your software, you’ll just end up hitting a wall faster than you ever have before.

What I’d like to see more of from others is exactly what I’ve tried to do here: honest, detailed accounts of building real software with these tools; not weekend toys or one-off scripts but the kind of software that has to survive contact with users, bug reports, and your own changing mind.

What Makes a Good Tool for Claude Code

Sun, 12 Oct 2025 00:00:00 +0000

I’ve been using Claude Code extensively for personal projects, and similar AI coding tools at work. Recently I came across this excellent blog post that resonated with a lot of my experience.

One part stuck with me though: Noah emphasizes that tools fail with LLMs when they’re “overly complex,” with the Unix philosophy being particularly well-suited for tool calling. But then I thought about git.

Git breaks the Unix philosophy completely. It’s sprawling, stateful, and complex. And yet Claude Code handles it effortlessly. It composes commands that, even after 10+ years of daily git usage, I wouldn’t think to use. It handles rebasing, cherry-picking, complex resets—stuff that trips up experienced developers regularly.

So if simplicity and the Unix philosophy aren’t the whole story, what else matters?

I’ve come up with three “hallmarks” of a good tool for tool calling with LLMs.

1. It’s been around for a long time and/or is used by lots of people

Examples: Unix tools like cat, sed, awk, grep, find—but also git, npm, docker, kubectl. Every Stack Overflow thread, blog post, and tutorial using these tools has likely ended up in the training data. Claude isn’t reasoning from first principles—it’s drawing on millions of examples.

This is why git works despite its complexity: Claude has effectively memorized decades of collective wisdom.

2. It has really good documentation (built-in help or external docs)

Even if a tool isn’t widely used, great documentation can bridge the gap. I’ve been building a finance system on top of Beancount—a double-entry accounting system that’s definitely not mainstream (maybe I’ll write a post about this in the future). Claude Code handles it surprisingly well because Beancount has exceptional documentation. When I point Claude at the docs, it can figure out the directive syntax, transaction formats, and account structures without necessarily having seen millions of examples in its training data.

Good --help text matters. Clear external documentation matters. If Claude can discover how your tool works, it can use it effectively.

3. Good error messages

Error messages, especially those with suggestions like “you used X, did you mean Y?”, can be tremendously helpful for LLMs. The best example is the Rust compiler: it gives errors like “you typed foobar, did you mean foobaz?” and Claude Code can actually use that feedback to correct itself.

This might be one reason why people feel Claude Code is particularly good at Rust programming—the compiler is essentially coaching it through mistakes in real time.

This isn’t to say the Unix philosophy is wrong — it’s that Unix tools work well with Claude Code for different reasons than we might think. Tools like sed and awk nail hallmarks 1 and 2: they’ve been around for decades (massive training data) and have extensive man pages (great documentation). The fact that they follow “do one thing well” is almost incidental to their success with LLMs.

So if I’m building a tool today, I can’t make it instantly popular, but I can make it understandable. That means good documentation, clear error messages, and a few solid examples of how it’s used. Those aren’t new ideas — they’ve always mattered. The difference now is that there’s much higher ROI in doing them than there used to be.

In other words, we used to write docs for humans. Now, we’re also writing for AIs that help humans. That shift changes the economics of good documentation in a big way.

Harnessing Frustration: Using LLMs to Overcome Activation Energy

Mon, 14 Jul 2025 00:00:00 +0000

One of my biggest weaknesses as a software engineer is procrastination when facing a new project. When the scope is unclear, I have a tendency to wait until I feel I’ve “felt out” the problem to start doing anything. I know I’ll feel better and work much faster when I get “stuck in” but I still struggle with that first step, overcoming the “activation energy” required to engage with the details.

LLMs have been a game-changer for me in this respect: I can just throw a couple of sentences at them with the shape of the problem. This leads to one of two outcomes:

The LLM comes up with a good solution, usually in a slightly different way than what I was thinking. I realize “oh wow the solution is much simpler than I thought”. Straight away I start thinking about the consequences of implementing and improving what the LLM suggested.
The LLM comes up with a solution that I intuitively recognize as “wrong”. My immediate reaction is frustration (“How could it get it so wrong”) which leads me to go back and forth with the model, explaining to it why its solution could not possibly work. But in the process of arguing with the model, my brain is churning away and generating variations or different approaches that could work. After a while, even if the AI is still on the wrong track, the debate will trigger a moment of inspiration where suddenly the solution will come to me. I’ll excitedly start up a new conversation and start working through it with the model.

The key is the emotional reaction I have immediately to the LLM’s response, either excitement or frustration. By harnessing this immediate feedback loop, I get my brain out of its passive, procrastination mode. It’s almost like a jolt: either I’m thrilled because it’s simpler than I thought, or I’m spurred to action by the urge to correct a perceived ‘wrong’ answer. This forces me to engage with the problem in a meaningful way.

For what it’s worth, this experience is very similar to talking through a problem with another engineer: the advantage of LLMs is that it’s available 24/7 and I never have to worry about my problem being “too insignificant” to bother someone with. The simple act of articulating my thoughts and hearing a response seems sufficient, regardless of the response’s quality.

So next time you’re procrastinating a problem don’t wait around for inspiration. Outsource the first step. Throw the problem at an LLM and see what happens. Whether excitement or frustration, by channeling that emotion you can solve your problem faster.