Chris Parsons

Webinar: AI Can Do (Most Of) Your Work

Thu, 08 Oct 2026 14:00:00 +0100

On 8 October 2026, I gave a webinar about which parts of your work to hand to AI. AI can now take a decent crack at almost anything you give it. In March I showed how I gave it all my context so it could. Since then I have learned that the harder question is which work is still yours.

One AI that knew everything

I run most of my business with AI. I had built up about 70 skills, nearly all working over one vault: every note, meeting transcript, saved article and client note, synced across my laptop, phone and a server. It was a thing of beauty.

So I asked it for everything. Client positioning, proposal wording, what to prioritise this week. It seemed to know my situation better than I did, and it never got lazy. What came back was plausible, and average. I read every draft, winced, and rewrote it. I had become a mindless human with a big rubber approval stamp, stamping out mediocre blog posts.

The thinking I had handed over was the work I am paid for, and the work I enjoy. AI is built to converge on an answer, and good strategy needs you to diverge first. One AI that sees everything is also a security risk, and blends client and personal signals into mush. I wrote about this in I Don’t Want an AI God.

We trust fluent writing, even when it is wrong, and the latest models write very fluently. Addy Osmani calls it cognitive surrender: the moment you stop forming your own answer.¹

What stays with you

Tara Seshan, who leads product for Codex and ChatGPT Work at OpenAI, names three things that stay with us: relationship and connection, expression, and accountability.²

Relationship and connection. My friend Hywel Carver has a good test: would you be comfortable telling the other person you used AI? Getting help appealing a parking fine is fine. A letter to your mum while she is in hospital is not. I use AI for quick “thanks, I would love to set up a call” replies all the time, but I always tweak them and I never let it send for me.

Expression. Some writing just carries information, such as a policy turned into a handbook. If the ideas are already formed, let AI write it. Other writing is your personal creative expression: your art, your personality on the page. That has to come from you.

Accountability. Sophie Alpert’s AI writing policy at Clay says to stand behind every line.³ “AI wrote that, just ignore it” is not an answer. If you could not be bothered to write it, why should anyone bother to read it?

Tobias Lütke calls unchecked AI text a slop grenade.⁴ One mistranscribed name in a meeting transcript gets picked up by other agents and spread across a whole organisation. So every transcript now goes first to an agent that checks it against my vault and asks me about anything odd, before other agents see it.

Goal, effort, check

Everything else I hand over, and it no longer takes clever prompts. Boris Cherny, who created Claude Code, said this week that prompts mattered a lot in the Sonnet 3.5 days. Now you tell the model what you want, how much effort to spend, and how it should check it did the right thing.⁵

The goal is the hard part, because often I do not know what I want yet. So I dictate, ramble, and ask the AI to question me one question at a time and keep its own view to itself. I want to work out what I think first.

This webinar is an example. At 9:30 that morning none of the slides existed. The AI pulled my notes from the vault and questioned me, then produced an outline. The flow was wrong, so I wrote the outline myself. The AI replied “that’s so much better than what I had” and rebuilt everything around it. Too right!

Then I give feedback on something I can see. My slides have a little review box where I comment on any slide, and the AI fixes everything in one go. You do not need my slide system. Claude artifacts and Codex sites make throwaway review pages in minutes, and they beat going back and forth in a chat box.

Writing

For about six thousand years, writing was proof of thinking. That stopped being true a couple of years ago. AI is brilliant at turning thoughts from one form into another, so it will happily turn a couple of thoughts into a whole blog post that still contains only a couple of thoughts.

One attendee asked: if your couple of thoughts are so easy to expand, did we need you to have them in the first place? Probably not. That post should have been a two-line LinkedIn post, or it needed more thinking.

So now the AI pulls up what I already know, and I write the argument myself. My first drafts are terrible. I do not enjoy it, but that is where the value gets added. Then AI tidies the structure, adds references, checks facts and points out holes in the argument. (I Think Whilst) Writing With AI walks through the whole process.

Coding

I delegate most of my coding now. I talk to one coordinator about what we are building, and it hands bounded jobs to worker agents and reviews what comes back.

My rule is to ask who will look at the result. If a test can check it, I delegate the whole thing. If a person has to see it, I build it interactively, often in a Claude artifact where I can comment on the screen.

One attendee asked about AI making huge “god objects” and five-thousand-line files. It happens. Good rules help, so does using only the latest models, and so does getting an AI to review the code adversarially. I care about the data model, the security model, overlapping abstractions and duplicated state, because they breed bugs, and about test coverage. I care much less about method length, or even which language it is in, because AI does the reading and writing.

When a game I was writing had a strange bug, I had the AI draw me the whole map instead of reading the code, and spotted which scenes were broken. (I Am No Longer) Coding With AI has more.

This is exhausting. I now spend the day on high-level judgement, across three or four projects where I used to run one. Another attendee explained why: AI does the fast System 1 work, so what is left for me is all slow, effortful System 2 thinking.

A team, not a god

For repeatable jobs, I have moved from one big AI to a team of small agents, each with one job. I have run them in Telegram for months, and I am now rebuilding them in Slack as 27agents. Em triages email, Miriam is a productivity coach, and Paolo is a product associate who also gets the code done. Each has a small remit, so you know what it can reach and how much damage a conversation can do.

The big players are reaching the same idea from different directions:

OpenAI dots gives you one primary dot that does everything. Single-job specialist dots exist only in enterprise pilots that OpenAI’s engineers set up with the customer.⁶
Grok Bot gives you a team of bots, but they all share one computer, and xAI says separate bots are not separate security boundaries.⁷
Claude Tag puts Claude into Slack, and the channel decides which systems and data it can reach.⁸
Shopify’s River is a Slack agent that only works in public channels and now co-authors one in eight merged pull requests.⁹

Most of these still leave you to work out what to ask and how to set up each bot. Shopify has a platform team do that for everyone, and so does 27agents.

A motor on each machine

Right now AI feels like the early days of electricity, with everyone wiring up their own generators and batteries at their desks.

Paul David’s “The Dynamo and the Computer” shows that factories which put one big electric motor where the steam engine had stood gained very little.¹⁰ They kept the same shafts and belts. The gains came in the 1920s, about forty years after the first power stations, when factories put a small motor on each machine and rearranged the floor around the work.

One big motor driving everything is the AI god. A motor on each machine is the team of single-job agents.

Electricity won when nobody thought about it any more. AI will get there the same way: small jobs, wired into the tools people already use, with a few specialists doing the wiring so everyone else does not have to.

Questions from attendees

One attendee asked how this works across a whole enterprise, and whether bots can be limited to certain jobs. They can. Copilot and Claude both have plenty of permission settings. Lock things down too far and you get generic answers, because the AI cannot see your data. Lock down too little and it runs riot.

Another asked how junior staff can reflect as carefully as seniors. Juniors cannot yet spot average output. AI raises their work to average, then hits a ceiling. They learn fastest working alongside a senior who points out the slop, until they develop a healthy slop allergy of their own.

Two more great suggestions came from the chat. Teach juniors, and children, to use AI to ask them questions rather than to give answers. And courses in systems thinking and critical thinking help. I have gone back to writing 750 words every morning for exactly that reason.

Key takeaway to remember

AI can do most of your work, but one AI that does all of it makes your work average and less safe. Keep relationship, expression and accountability for yourself. Hand the rest to small agents with one job each, and spend the time you get back thinking.

One thing to try this week

Take a week of your work and sort each task into three piles. Information: reports, summaries, admin. Ideas you have already formed that just need writing up. And you: relationships, original thinking, anything you have to stand behind. Hand over the first pile, let AI help with the second, and keep the last for yourself.

Tools and resources

The slides from this webinar
I Don’t Want an AI God, on why one assistant with access to everything goes wrong
(I Think Whilst) Writing With AI, my full writing process
(I Am No Longer) Coding With AI, how I delegate code
27agents, the team of single-job agents I am building

Addy Osmani, Cognitive Surrender. ↩
Tara Seshan on Lenny’s Podcast: AI’s third era: the rise of persistent AI coworkers. ↩
Sophie Alpert, Clay’s AI writing policy, 19 August 2026. ↩
Tobias Lütke on The Knowledge Project podcast, as reported by Fortune: Shopify CEO says employees’ ‘slop grenades’ are making more work for everyone else. ↩
Boris Cherny on X, 6 October 2026. ↩
OpenAI, Introducing dots. ↩
SpaceXAI, Grok Bot overview. ↩
Anthropic’s Claude Tag announcement, 23 June 2026. ↩
Shopify Engineering, Under the River, 2026. ↩
Paul A. David, The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox, American Economic Review, 1990. ↩

(I Think Whilst) Writing With AI

Fri, 02 Oct 2026 08:00:00 +0100

For thousands of years we have treated writing as proof of thinking. You could not produce a fluent, well-argued piece without doing some thinking along the way. That is no longer true.

Large language models (LLMs), the engines underneath most AI agents, are born to write. Writing is how they reason about the problems we give them, and it is how they talk back to us. Fluent words now cost almost nothing, and we have to build a new mental model of writing: words do not mean thought.

When people use thought-free output verbatim, the result is slop. LinkedIn shows the problem well: since ChatGPT launched, the average long LinkedIn post has more than doubled in length.¹ I very much doubt the thinking inside those posts has doubled too. Worse, slop does not just sit there. People read half-formed, average ideas, copy them into their own notes and repeat them, and the research suggests that AI editing shifts what people conclude, not just how they phrase it.² Ideas get diluted and proper thought gets stifled, the vicious cycle I wrote about last year. Shopify’s CEO Tobias Lütke has a name for the workplace version, slop grenades: AI-generated emails and code that people toss at each other without reading or understanding them.³ The author saves time, and their colleague has to do the thinking instead.

How do we get out of this mess? By remembering that writing was never just about producing words. The words are the output. The thinking we do while we write is what matters, and readers can tell when it is missing. This article is about how I use AI to write, and how I try to let it speed me up without skipping the part that adds value.⁴

Writing Is Where Thinking Happens

Some writing is slow and painful. It is the forging and fusing of our thoughts with the words that appear in front of us. It takes honing, editing, and ruthless cutting down to the most succinct version of what we think. The product is a polished set of thoughts that adds something to a debate. Tara Seshan, who leads product for Codex and ChatGPT Work at OpenAI, calls this writing to think, as opposed to writing to report.⁵ When you write to think, the writing is where your position gets formed. When you write to report, you are converting thoughts that already exist from one form into another, such as code into documentation, or a set of ideas into a LinkedIn post.

AI is brilliant at converting thoughts and very bad at coming up with original ones. The danger is that it converts so well that you do not notice the original thinking has been skipped.

I have a confession about this. I caught myself doing exactly this in my early days of writing with AI, because I had not really thought through the difference. I would dictate a stream of consciousness, let the AI polish it up, and send it. But a stream of consciousness is not a well-formed thought. What came back was 90 percent good enough, but it was not any good, because it was not my formed thoughts. Some of my earlier articles are scattershot and do not always hang together. They were first drafts, polished and tidied to look like more than that (I have since cleaned up a lot of them on this site, but you might still find some if you look hard enough!) Polished unformed thoughts are the worst kind of writing, because the polish hides the gap from the reader and from you.

This is also why slop feels so empty to read. We are wired for story and meaningful connection, and the content people share most is the content that makes them feel something.⁶ When we read, we feel connected to the writer’s inner world, and we process our own thoughts alongside theirs. If we sense that an AI was involved to the point that the words are no longer the author’s creation, we feel flat and hollow, as if there was nobody on the other end.

When AI Should Write It

This does not apply to all writing. A letter to the council complaining about a parking fine is very different to a quarterly strategy.⁷ What matters is what the writing is supposed to carry, and whether that part has to come from you.

Some writing only carries information. Translating a document, turning code into documentation, or reshaping a company policy into a handbook all move meaning from one form to another. That is barely writing at all, in the same way that heating something up is not really cooking.⁸ If no original thought happens in the process, it is a waste of time not to use AI.

Some writing carries ideas you have already formed. I write a lot of posts on LinkedIn, and I do not mind AI helping turn a set of ideas I have already thought through into another format. The infographics I make for my posts are the same: they convert an argument I have already written into a picture, so I am happy for AI to draw them. Comics are a bit different. A good comic makes a joke or lands a point in a single frame, and getting that right is thinking in its own right. I often go through fifteen rounds of revision before one works, and I keep the whole provenance chain for each one: every prompt, every rejected draft and every correction I made along the way.

That does not change the fact that there is a huge amount of slop on LinkedIn. The slop is created too quickly by people who haven’t thought through what they’re saying. I do not care who wrote the words, but content built on ideas people have not come up with themselves adds nothing. It is just noise. The same goes for comments. Automated commenters are farming engagement, and they produce the worst kind of slop.

Some writing carries you. Creative writing that weaves story or meaning is an expression of self, so you need to have thought through the content and stand behind the meaning and intent of every word. Writing to build a relationship goes further still. If I am trying to connect with someone, or tell them how I feel, there is no place for an LLM in forming those thoughts. There is a big difference between a quick follow-up to schedule a call and a letter written from a place of deep feeling. I have no problem with AI doing the first. Using it for the second tells the other person I do not care about them. There is also a difference between using AI to send cold emails at scale and sending a thoughtful note about something that made you feel a connection with someone. That second note is not logistics. You are trying to build a personal connection, so write it carefully. At the very least, let the AI give you the bones and edit them until they are really yours. My friend Hywel Carver has a good test for this: would you be comfortable telling the other person you used AI? If not, that discomfort is your empathy talking, and you should probably write it yourself.

How I Write Now

What I care about is whether people did the thinking, and whether they meant it when they claim to be expressing themselves or connecting with me. So when a piece needs real thought, or someone is paying me to think through a problem, I make sure I go through that process myself.

I start by asking an AI to pull up everything I already know about the topic. I keep my notes, meeting transcripts and saved articles in a folder my AI can read, so it can find what I said in a meeting three weeks ago, or the voice note I recorded after listening to a podcast. It gives me that back as bullet points. I had been collecting thoughts for this post for a few months, so it surfaced meeting notes, voice notes arguing both sides of the question, and articles I had saved along the way.

Then I hold as much of that in my head as I can, and write the argument myself, either typed or dictated. When I do want help with structure, I ask for an outline rather than a draft. Something like: “From this, give me a structure. Do not write the prose. List the argument, the open questions, and what is missing.” Then I write it up myself. Doing a comic first is a useful forcing function. My comics tend to be good precisely because coming up with the premise forces me to decide what I actually think.

The thinking does not always have to happen at the keyboard. Sometimes it happens in conversation. If I have already worked through a set of changes with someone on a call, I am happy for an AI to apply them from the transcript. Making all those edits by hand is only worth it if I want to think them through again.

Once I have a draft, I refine using an AI. I ask the AI what it thinks my argument is. On the first draft of this post, it told me the title promised one argument and the body made another, and that my best paragraph was buried at the bottom of a list, and it was right, so I rewrote the structure and only then did I let it help tidy the prose. I fact check and chat to the AI about various sections or things that feel “off”.

Finally, I read every line and ask myself whether I stand behind it. Some writers go further than this. Thomas Ptacek will not use a single word an LLM suggests, because the phrasing carries a style you can smell.⁹ I can see the appeal, and it is a reasonable rule if you want it. My line is looser: the ideas, the evidence and the judgement have to be mine, and I need to recognise the meaning of every sentence.

Keep the Receipts

There is a legal reason to do the thinking yourself, too. Copyright law protects human authorship, and it is not yet clear how much of an AI-assisted piece counts. The US Copyright Office says prompts alone will generally not make you the author, but your own selection, arrangement and editing of the output can be protected.¹⁰ In the UK, the law has a rule for computer-generated works that predates generative AI and has never been properly tested on it.¹¹ If the machine did all the creative work, you may not own the result in any meaningful way.

That is why I keep provenance records for my comics and infographics. The record is not a guarantee of copyright, but it is evidence of what I contributed: the idea, the choices, the corrections and the drafts I threw away. The same logic applies to writing. If your first draft, your outline and your edits are your own, you can show where your thinking went in. If all you have is a prompt and a polished output, you cannot. (This is practical information, not legal advice.)

Stand Behind Every Line

Companies are starting to write these policies down. The best AI writing policy I have seen is Clay’s, written by Sophie Alpert.¹² It has four principles:

You must stand behind every idea and every sentence in your docs, and “AI wrote that, just ignore it” is not an acceptable answer.

Writing is thinking, so outsourcing it to skip the thinking defeats the point.

More time should be spent writing a document than reading it, because one author’s effort saves time for many readers.

Longer is not better. You have probably heard that Blaise Pascal once apologised for a long letter because he had not had time to write a shorter one.¹³ AI writes the longest version of all, in the quickest time.

The policy even allows quoting AI directly, as long as you say so: “Claude suggested this, what do you think?” is honest and useful.

I would add one thing from my own experience: develop a slop allergy. People who are new to AI often cannot spot content free writing, because it looks so polished. With practice you start to see the shape of it before you have read the words. Get to the point where you would be embarrassed to send it.

Make sure your team are thinking, the simplest way to do this is to ask them what they think about a piece of their writing, and how did writing this help you get there? It will become clear quickly whether the thinking has been done. Do not judge people for using LLMs to write. They might be tidying up their English, or their spelling. Judge the thinking behind the writing on its merits.

Fluent words are free now, but thinking is not, and thinking is still the reason anyone reads what you write. AI is brilliant at converting thoughts and very bad at coming up with original ones, so give it thoughts worth converting. Do the thinking yourself, let AI help with the rest, and stand behind every line.

Originality.ai examined 8,795 long LinkedIn posts (100 words or more) from 2018 to 2024 and found a 107 percent jump in average word count since ChatGPT launched, as reported by The Register. ↩
The LLM writing distortion study found that when models revise human writing, they shift the argument and its conclusions, not just the style, and push everyone in the same direction. LLMs homogenise writing. ↩
Tobias Lütke on The Knowledge Project podcast, as reported by Fortune: Shopify CEO says employees’ ‘slop grenades’ are making more work for everyone else. ↩
I wrote the first draft of this article entirely by hand, to work out what I think about the topic, AI helped me tidy it up. The thoughts are my own and I stand behind every word. More on how below. ↩
Tara Seshan on Lenny’s Podcast: AI’s third era: the rise of persistent AI coworkers. ↩
Jonah Berger and Katherine Milkman studied around 7,000 New York Times articles and found that content which stirs high arousal emotions, such as awe, gets shared most: What Makes Online Content Viral?, Journal of Marketing Research, 2012. ↩
The Economist makes a similar contrast between contesting a parking fine and writing an apology: AI often makes writing worse. (Even if it makes it better). ↩
If you want to do some proper cooking rather than heating things up, Cherrypick plans your meals and fills your supermarket basket with everything you need to cook them. I co-founded it, so I am biased :) ↩
Simon Willison quoting Thomas Ptacek: How to write with an LLM. ↩
US Copyright Office, Copyright and Artificial Intelligence, Part 2: Copyrightability, January 2025. ↩
Section 9(3) of the Copyright, Designs and Patents Act 1988 says the author of a computer generated work with no human author is the person who made the arrangements necessary for its creation. ↩
Sophie Alpert, Clay’s AI writing policy, 19 August 2026. ↩
Often attributed to Oscar Wilde or Mark Twain, but it was Pascal, in the sixteenth of his Provincial Letters (1657): “I have made this longer than usual because I have not had time to make it shorter.” ↩

Ship Like an Engineer Without Writing Any Code

Thu, 24 Sep 2026 13:00:00 +0100

This post is based on a workshop given on 24th September 2026 at design+AI.

Building a working prototype with AI is easy now: ask Claude to build you something cool and it will. Shipping that prototype, so that it is live and useful and does not fall over, is a different problem. On Thursday I spent a morning in Brighton with a room of designers and product people working through the practices engineers rely on to make that second part possible. We did not write any code ourselves. The agent did all of that. Our job is simply knowing what to ask for.

There is a mystique around engineers and how they work, but we are not always that clever, and our whole job is managing complexity. The systems we build are too complicated to hold in our heads, so over the years we have built up habits that let us keep going without everything falling over. I think of them as the foundations under a building, without which you can only go so high before it collapses. A lot of these habits are about being lazy in a disciplined way, so that you do not have to think about the same thing twice.

That matters even more now. I do not know any engineers who still write code by hand, because they tell the AI what to write. Agents are good at making sensible decisions, but they still need nudging, and whether you type the code yourself or tell an agent to type it, you have to know the important things to say so that it does not make a mess.

We built a small lighthouse adventure game (we were in the Lighthouse centre, after all) in Claude Code or Codex in the terminal. The terminal gives the agent the most access to your machine and your files, so it can do the most for you. Then we worked through nine steps.

1. Start With What People See

The first step was to build a prototype and play with it. Engineers call this working outside in: you start with what the customer sees and work back towards how it works underneath. People in the room asked for more rooms, or for SVG pictures, or for new fonts, and each change took moments. When you start a new app you often have little idea of what you want, and you need to play with it to find out. Doing that while it is small is far easier than doing it later, and it saves you building a large system underneath an interface that turns out to be wrong.

I asked everyone to tell the agent to keep it simple at this point. We were only prototyping, and a prototype that has already grown into a full app loses the advantage of being cheap to change.

2. Save Every Version

If you have ever saved a file called “presentation V2 final final”, you have already used a form of source control, because you were saving successive versions so that you could always go back. Engineers do the same thing with code using a tool called Git, and GitHub is the website where you publish those versions and share them with other people.

Git is hard to learn, with dozens of commands that even developers find confusing, but you no longer need to learn it. Claude knows Git far better than I do, so you can just ask it things: show me the last few changes, undo the change from three changes ago, take everything back to how it looked this morning, try this idea on a separate branch. I commit every five minutes or so, whenever I make a change, and I let the agent write the commit messages. If you want to understand something, keep asking the agent to explain it. Staying curious is probably the most important thing to take away from the whole session.

3. Deploy on Every Push

The third step was the hardest. We set up automatic deployment, so that every time you push a new version to GitHub it goes live for everyone within a minute or two. That sounds terrifying at first, but there are ways to stop a push from breaking things, which is what the later steps are for. The reason to set it up now is the same as before: deploying a tiny app is fiddly, but deploying a huge app that a team has spent six months building is much harder, and nobody can see any of the work until you do.

Setting this up meant dealing with secrets: the tokens that let GitHub tell the hosting service to publish your app. Secrets never go into your code, and instead live in a locked store that only the deployment process can read. This is one of the rules from the Twelve-Factor App, a standard developers use, and the test it gives is whether your code could be made public at any moment without leaking any credentials.¹ When I worked on GOV.UK early on, the source code for sites like Number 10 and the Ministry of Defence was public for anyone to read, and it was still secure because nothing secret lived in the code.

One attendee asked a good question here. Claude would not let them paste a token into the chat and gave them a script to run instead. They could not read the script, so which was safer: running a script they did not understand, or pasting the secret into a chat that is not published anyway? My honest answer was that Claude is doing the right thing, because it has been trained not to hold your secrets, but it is not infallible. A model can be manipulated, or it can get part of a script wrong. In practice I have not seen it do anything stupid for a long time, and I trust it more now than I did at first, but I cannot promise it is always right, because I do not control the models. When you are unsure, this is a perfect moment to message a developer you know and ask whether the script is safe, which takes them five minutes.

4. Choose the Framework Early

Next we turned the prototype into a real app using Next.js. I pick Next.js because it is so widely used that agents know it better than almost anything else. It is not my favourite framework to code in, but I do not code any more, so I do not mind. A framework gives the agent a structure: it knows where the code goes, where the assets go and how everything fits together. Without one, the code gets messier as it grows, and you find yourself asking the agent to clean it up again and again. When Next.js first came out, this conversion would have taken a mid-level developer half a day or more, and the agent did it in minutes.

Choosing a framework is harder to undo than most changes, though. If you build on it for a few days and then decide you want something else, that is a lot of work to unpick. A few decisions are like pouring cement: the framework, the database, how people sign in and how your data is managed. Those are worth talking through with the agent before you start. Should this be a mobile app? What about accessibility? What is the difference between React Native and Flutter? The agent knows the trade-offs as well as any senior developer, and you do not need to be one to ask. People ask me about spec-driven development, where you write a detailed specification for the AI first. I am not a fan of formalising that too early. A thoughtful conversation with the agent, followed by asking it to write down what you agreed, is usually enough. There is no point getting it to write a big spec if you are not going to read it.

5. Let the Build Say No

Once every push goes live, you need a way to stop a broken change from reaching people. So the fifth step added automated tests and a build. Tests are code that runs against the app itself, rather than for your customers, to check that it still does what you expect, whether that is a sum coming out right or the arrow keys moving you between rooms. When you push to GitHub, the build runs all the tests, and if any fail the change never goes live.

The game used the arrow keys, so once it was deployed a couple of people opened it on their phones and found they could not move at all, because phones have no arrow keys. That was a useful accident! Things you did not expect turn up as soon as real people on real devices touch what you built. It is exactly the kind of mistake you can ship without noticing, and then you are fixing it while the support requests pile up. (I have been in a room full of developers fixing a system while it was being announced on the BBC, and I do not recommend it.)

I asked the agent to write each test before the code, which is called test-driven development. It writes a test for the new behaviour, runs it and watches it fail, then changes the code until it passes. Writing the test first forces you to decide what the code should do, and it lets you build in small slices. Not everyone agrees with writing the test first, but everyone agrees you should have tests, and putting them in early means you start as you mean to go on.

We then broke the build on purpose, by telling the agent to change the code and leave the test alone, then push it. GitHub showed a red cross next to the change and nothing went live. One attendee’s agent refused to do it at all, because the instructions contradicted the test. The rest did as they were told, which is a reminder that an agent will usually follow a clear instruction even when it is a bad one. A red cross only protects you if you fix it straight away, which is why I wrote back in 2012 that you should never leave a failing test.

6. Make Mistakes Harder

Four plus four is eight, but the text “4” plus the text “4” is “44”, because adding two pieces of text just joins them together. Computers treat numbers and text differently, and JavaScript, the language of the web, tries to be clever about guessing which one you mean. In a big program, that cleverness makes it hard to work out what is going on. TypeScript makes the types explicit, so mistakes show up early. The easier it is for a person to reason about the code, the easier it is for an agent too. If you are not a programmer, the advice is short: tell the agent to use a typed language (TypeScript is a good default if you are building for the web).

Linting is the tidying part. I admitted I did not know where the name came from, and one attendee suggested a lint roller, picking the fluff off your code, which is as good an explanation as any. A linter checks that the code is consistent and well formed and flags anything left lying around, such as code nothing uses any more. Developers used to argue endlessly about whether to indent with tabs or spaces. In the end, bored CTOs handed the decision to the linter, and nobody has to care any more. Lint and type checks go into the build alongside the tests.

7. Write Your Decisions Down

By this point we had made a number of design decisions about the codebase: Next.js, strict TypeScript, tests first, linting. Other people would choose differently, so the agent needs to be told. That is what an AGENTS.md file is for²: it sits at the top of your project and the agent reads it before every session. There is no point writing “be a really good programmer” in it, because that is already built in. Use it for your particular decisions and context: what the app is for, who it is aimed at, where your style guide lives, and rules like “commit after each small change” or “ask me before adding libraries”.

8. Build Features Quickly

With the foundations in place, I gave everyone ten minutes to build the most impressive thing they could. Once the scaffolding is there, you can add a feature every two or three minutes. My favourite technique for this is to say “ask me one question at a time until you have enough information”. If you want multiplayer but know nothing about it, the agent will ask whether you want turns or real time, then whether there is a timer, and it keeps going until it can build what you want. I use the same questioning approach when coding, as I described in my Ditch Autocomplete talk.

9. Data Does Not Roll Back

The last step moved the game’s rooms out of the code and into a database. A database looks a lot like a spreadsheet, with stricter rules, and it can grow to billions of rows. Storing the rooms as data means you can change them without changing and redeploying the code, and you can track players and where they are.

Data brings its own risks, and I used the recent NATS air traffic control failure as an analogy. The faulty software was rolled back quickly, but the disruption took days to clear, because planes and crews were all in the wrong places. Code is like that software: you can roll it back fast. If a bug has sprayed bad data through your system, cleaning it up can take far longer. Changing the shape of the data, such as adding a table or renaming a column, is done with a migration: a small file in your project that describes the change and runs as part of the build, and the agent writes it for you. If you store anything about people, you also need to think about regulations like GDPR.

If any of this gets too technical, ask the agent to explain it again in simpler terms. I often ask it to explain things as if I were 15. (I tried asking it to explain things as if I were five, but I got a lot of fluffy bunny analogies.)

Call an Engineer For These

AI can do a huge amount of this on its own, but the more complex your app gets, the more you will need to talk to a friendly engineer. Sign-in is easy to get wrong, so use a provider like Auth0 or Supabase rather than having the agent build its own. Do not have it write anything that takes credit card details either, and let Stripe handle payments. If you are storing personal data in any professional setting, compliance matters, and it is worth checking that you are storing it securely. Before real people rely on your app, and certainly before you charge them, get someone to review it. The same goes for any data migration that makes you nervous, and keep an eye on your cloud bill, because people have had some nasty surprises. You could ask another agent to check your work, and it would probably do a good job, but an agent cannot be accountable, and asking someone to check is a decent thing to do.

Engineering Is Unlocked

Engineering is not scary, and it is not out of reach. None of these practices needs you to write code. They are things you can ask an agent to set up for you today.

Take a prototype you have already built and open it in Claude Code or Codex. Ask the agent to put it in Git, publish it to GitHub and set it up so that every push deploys it automatically. Then ask it to explain each step it took, one at a time, until you could explain it to someone else.

The agent writes all the code now, but you still need to know what to ask it for. Engineers have spent decades building habits that make complicated software manageable: small versions you can rewind, deployment on every change, tests that stop a bad change going live, and data you treat with care. You need to know those habits exist, ask the agent to set them up, and keep asking it questions until you understand what it has done. I go further into how I work with agents in my post on coding with AI.

III. Config, The Twelve-Factor App. ↩
Claude Code used to read only CLAUDE.md, but Anthropic added AGENTS.md support on 18th September 2026. In a project with no CLAUDE.md it now reads AGENTS.md, so one file works for Claude Code, Codex and most other agents. ↩

Time To Hire An AI Head

Thu, 24 Sep 2026 09:00:00 +0100

For a long time I have wanted to hand AI a senior job. When I was a CTO, this was the kind of work I would have delegated to a head of department: take a loosely described outcome, work out what it really needs, split it up, get other people to do the pieces, and come back with something finished while I got on with my own work.

That goal has been elusive for years. Until very recently, the best description of an AI model was an overeager junior. Delegating to one meant giving it lots of context, and then watching it closely. That was fine, but frustrating.

Working with a junior model came with a lot of approval steps, because I could not trust it. Every run needed me to check the plan, answer its questions, approve its commands and review what came back. Cory Doctorow has a name for this: the reverse centaur: a human used up by the machine, pressing the buttons it cannot press itself.¹ I built a whole review board to cope with my agents’ output, and found I had turned myself into the button-presser, which I wrote about back in March. A week later I was warning that the 85th correct approval trains you to blindly approve the 86th, and by May I could see that the board itself was training me into drive-by decisions. (See anti-pattern 2 below.)

Fable and Astra changed that. After reading Steve Yegge’s essay on the shape of things to come,² I moved to Astra, stepped up a level and stopped managing individual tasks. Astra in particular is genuinely more than a senior: it behaves like a Head. I can give Astra or Fable a high-level instruction and it will just crack on: it reads what it needs, splits the job up, hands the bounded pieces to cheaper workers and comes back with something finished.

This all works very well, until the tokens run out. Using Astra or Fable as my main conversation partners burns through even the top subscription plan at a frightening rate, and I do not want to spend one every day. So some finesse is needed: I want to delegate at a high level without paying Head rates for every piece of the work.

Something else has happened at the same time: the open models have caught up. The previous open models I tried were straight up incompetent at anything beyond a narrow script, but GLM 5.3 Flash is a competent junior.³ It has a stab at most things, including coding, and runs most of my agent fleet for pennies. On its first day as the standard model for my agents it handled 32 jobs with no failures and no invented facts. It still often misses things or does not quite finish, like any junior, but it has matured enough to be useful and does not break the bank.

So I now have expensive Heads who can take a whole outcome, and cheap juniors who can do a lot of the work. Here are the patterns I have found for combining them.

Pattern 1: Put A Head In Charge

You

outcome

one finished result

Head

briefs

results to review

Junior

This is the pattern I use most. I give the Head an outcome, it works out the approach, and it splits the job into bounded pieces that cheap juniors can finish. It briefs them, resolves the dependencies between them, reviews what comes back and owns whether the whole thing is accepted. I have a delegate skill that manages this for me, choosing the right level of model depending on the task. Stephen Bungay calls this middle layer operations in The Art of Action: strategy sets the intent, tactics carry out the tasks, and operations turns one into the other.⁴

Across three matched coding tasks, GLM 5.3 Flash cost about $0.61 against $16.95 for the flagship model from the same family. If you want to use these more junior models, you need to make sure something competent catches what they miss. In one audit Flash built a feature that forwarded messages between channels. The tests passed, but the code did not do what I had asked for when the original channel could not be read, because the test only checked a weaker fallback. The real fix is an acceptance test that checks the actual requirement, and the Head should insist on one before it accepts the work. Part of that failure was mine, too: the reviewers I brought in afterwards were never shown the original brief, so they judged the work against the wrong standard. The Head needs the original request, the boundaries and the evidence, not just a message from the junior saying it is done.

(Anti) Pattern 2: Do Not Manage The Juniors Yourself

Junior

???

context, every time

Youmanaging everyone

???

context, every time

Junior

When the juniors got good enough and my Max plans were exhausted, I spent a few days last week handing them work directly. This was not good: it took me straight back to the reverse centaur and the bad old days of 2025, when I was testing coding agents head to head and concluding that independent coding agents were not ready. I found myself shaping my work around what they could do, deciding the pieces, checking for omissions, answering their questions and keeping everything moving. Worst of all, I had to give them loads of information every time, because a junior only knows what you tell it.

A friend asked me how you give a senior model enough context to do the job the way you want it done. For a junior, that really is the hard problem, but for a good Head, it mostly goes away. A good Head fetches its own context and works things out from the wider picture, which in my case is my vault: years of notes, projects, decisions and skills describing how I like things done. In an organisation it might be the repo docs, a wiki or the ticket history. If the context is not written down anywhere, the Head cannot fetch it, so writing it down is the first job. I do not need to give it anything more than that, and the juniors get a tight brief from someone who understands the whole job. The answer is simple: only talk to really good agents, and let them delegate the rest as in pattern 1.

Pattern 3: Give The Head Room To Think

Headlow effort

same model, more thinking

Headhigh effort

There are two choices hiding inside “which model”. The first is capability: have I picked someone who can do the job at all? The second is how much reasoning effort I let it spend. A Head at low reasoning feels like a senior person on too little sleep and too much coffee. They are still senior, but they move fast and miss things. At high reasoning it is the same person, rested and with an afternoon clear. Setting the two separately saves a lot of tokens: medium effort for day-to-day coordination, and high effort only when a job needs real thought. How much it saves depends on the model. On Anthropic’s own coding benchmark runs, Opus 5.5 at medium effort scored about 2.5 points below high for about 70% of the cost, and extra-high effort bought only 1.4 more points for two and a half times the cost of high.⁵

Pattern 4: Visit The Expert In Their Den

Younot involved

Head

problem, evidence, doubt

second opinion

Expertincomprehensible, annoying

Some problems deserve a second opinion. My Head can run at medium effort and call in a fresh expert at high effort, either another copy of the same model or a different one entirely. The fresh expert gets the original problem, the evidence and the specific doubt, with none of the baggage of the conversation so far. The Head weighs what comes back and stays responsible for the result.

I used to do this with Fable. It felt like visiting the brilliant expert that every organisation has, sitting in their office den: you go occasionally, and you dread the experience, mostly because of what Ethan Mollick calls “Claudish”, the nonsense-speak these models drift into.⁶ Opus 5 had the same problem, which is why I never used it as a Head. The advice was excellent, but I did not want to deal with it directly all day, and I certainly could not afford to use a top-end model every time: the plan usage was just too great. So I sent the Head to visit on my behalf, and Fable only ever saw a well-prepared question. Now that Opus 5.5 is my go-to Head, I make that visit far less often. I suspect the den will come back, though. In future there could be specialised and very expensive models built for exactly this kind of work, and the pattern will be the same: prepare the question well, send the Head, and keep the expert out of the day-to-day.

Pattern 5: Try The Junior Head

Junior everythingGLM 5.3 Flash

sweet spotJunior HeadOpus 5.5

Head of everythingAstra, Fable

Capability

Cost

This is the newest pattern, and the one that makes the whole arrangement affordable: run most of the work through a cheaper Head, and escalate to the expensive one only when a job needs it. For me, that cheaper Head is Opus 5.5, which seems to have fixed the Claudish. It is proving to be the first model I can use as a Head without watching my subscription drain. It is capable, it is fast and it is almost as good as Astra at strategic work. I am also not using it to delegate as much, because it can do the work itself at a reasonable cost. I think of it as a junior Head, or a Head who has almost got it. Astra is still my head of everything, but Opus 5.5 runs the day-to-day.

Because it does not burn through my tokens, it widens what I can experiment with. I can hand it outcomes I would never have risked on Astra’s budget, see what comes back and learn from it. When a job turns out to need more, I escalate to Astra or send the junior Head to visit Fable.

Pattern 6: Ask The Head To Explain Strategy

Vaultnotes and archive

reads it all

Head

explains

Short brief

You decide

In May I wrote that I did not want an AI god, because asking AI for a strategy gets you something “plausible, coherent and mind-numbingly average”. I still believe that about asking a model to decide for me. What has changed is how good the Heads have become at helping me understand a strategic situation. Increasingly I ask them to read everything relevant and give me a good understanding of where things stand, and that is proving very helpful. The call is still mine, but I make it with a better picture than I could have put together myself in the time.

Understanding a situation is different from interpreting it and expressing what to do about it. Tara Seshan put this well on Lenny’s Podcast:⁷ as AI takes on more of the work, three things stay human, and they are expression, accountability and relationships. The Head can help me understand the strategy, but saying what we will do, standing behind it and bringing people with me are still my job.

Head Of Everything

I am not the only one reaching for the org chart. Every’s Context Window newsletter reported engineers describing Anthropic’s models as a company: Fable as the CEO, Opus 5 as a senior engineer and Sonnet 5 as a junior engineer or analyst.⁸ The same piece has a theory for the Claudish, too: that Opus 5 was trained mainly to work as a subagent under Fable, so its prickly style matters less because it mostly talks to another model.

Where I differ is on how far to take the chart. I do not think giving agents human job titles works well. Agents have properties that humans do not, which are genuinely useful, and mapping them onto a human role can limit them. It also suggests that AI is a direct replacement for a person, which is wrong: AI takes on slices of people’s work, not whole jobs. Human job titles are far too generic for that anyway. The best title for an agent is one you never see in the real world, like “financial reconciler” or “productivity coach”: a single slice. The level you give it, junior or Head, then depends on how powerful the model is and how much you want to spend.

This will likely change in future. The same piece reports the labs moving towards specialised models, with one to orchestrate and another to execute. We may yet end up with truly specialised intelligences, and who knows where that goes. For now, the level is what I notice first: they are good at everything.

The employee metaphor is not perfect, but where it breaks, it mostly breaks in the agent’s favour. An agent can be made to forget things completely, which would get very strange in a human (Eternal Sunshine of the Spotless Mind⁹ is a fantastic thought experiment on exactly that!). And we never hire a junior model and promote it: we pick the right level for each job, so the Peter Principle does not apply.¹⁰

Buy Back Your Attention

All of this comes back to the goal I started with. With a team of juniors, I end up squeezing every last drop out of my subscription while spending the attention I was trying to free up. With a capable Head, I can say “please get that done” and go away for hours. So I have stopped comparing models and started comparing teams: the whole cost of an accepted outcome, including the briefing, the review, the repairs and above all my own time spent stepping in.

If you are putting together a team of models, start with the management questions rather than the benchmark tables. Who understands the whole task? What can each worker reliably own? Who fetches the context? What evidence proves the work is done? What can the Head read, and where does that data go? The juniors are now good enough to do a lot of the work. They still need somebody to manage them, and, for many tasks, that somebody no longer has to be you.

Cory Doctorow sets out the reverse centaur in his speech on the AI bubble: a centaur is a human helped by a machine, and a reverse centaur is a human used up by one. ↩
Steve Yegge, The Shape of Things to Come, Part 1: The Continuous Thunderdome, August 2026. ↩
The next one I am keen to try is Xiaomi’s MiMo-V2.6, released with open weights on 21 September 2026. My setup only uses hosts that offer zero data retention (ZDR), and there are not enough of those for MiMo yet. ↩
Stephen Bungay, The Art of Action. His chapter summaries set out the strategy, operations and tactics distinction. ↩
Anthropic, Optimizing for cost and intelligence, SWE-bench Pro results. The same page shows research tasks where higher effort bought no measurable accuracy at all. ↩
Ethan Mollick coined the term on X in June 2026, watching long agent tasks drift into a private dialect of labels, fragments and shorthand. ↩
Tara Seshan on Lenny’s Podcast. ↩
Every, Fable as CEO, Context Window newsletter, July 2026. ↩
Eternal Sunshine of the Spotless Mind (2004), written by Charlie Kaufman and directed by Michel Gondry. ↩
Laurence J. Peter and Raymond Hull, The Peter Principle (1969): in a hierarchy, people tend to be promoted until they reach a role they cannot do. ↩

Call It Machine Intelligence

Mon, 21 Sep 2026 08:00:00 +0100

Donald Trump wants a better name for artificial intelligence. On Saturday he called the words “inaccurate, and very ineloquent”, and asked his followers to vote between Superior Intelligence, Extreme Intelligence and Supreme Intelligence.¹ He is right that the name is wrong, and I have a better option than any of his three.

Let’s call it Machine Intelligence.

McCarthy Wanted a Better Name Too

John McCarthy coined the term in his 1955 proposal for the Dartmouth summer workshop, and he later told Noel Sharkey that he wished he had called the field “computational intelligence” instead, although he accepted that his choice had probably drawn more people to the subject.² The field has also dropped the name under pressure before. When the funding dried up, researchers called their work informatics, knowledge-based systems or cognitive systems to keep the grants coming, and IBM later sold Watson as “cognitive computing” to get away from the fear attached to AI.³ None of those names changed how anything worked, and none of them stuck.

“Computational” is no better than “artificial”, because it is long and awkward to say. “Machine” is clearer and more visceral, and it will get more accurate, because this intelligence will be embodied in robots before too long. How soon depends on whether we get export controls or a deal, and I wrote in April about how governments take control of a technology once the political panic sets in.

Artificial Implies Fake

“Artificial” implies that the intelligence is not real. Biology has had the monopoly on intelligence until now, so we have conflated the two for our entire existence and built our assumptions on top of that. That monopoly is over. I suggested Machine Intelligence in a WhatsApp group with other AI experts, and I have not heard a better name. It covers any intelligence produced by machines, and says nothing about real or fake or about biology. The term has a history of its own too: Donald Michie convened the first Machine Intelligence Workshop at Edinburgh in 1965, and the university founded a Department of Machine Intelligence and Perception the year after.⁴

Trump has more important work to do than naming things. He could make an information-sharing deal with China that lets them catch up to the level of OpenAI’s Astra, in return for slowing further work.⁵ UN inspectors would get access to AI labs worldwide, and he could share a Nobel Peace Prize for it. That deal matters far more than what we call any of it. But he is right about the name, and “artificial” has to go. Once it is gone, we can no longer get away from the fact that machine intelligence is real, and here to stay.

Trump posted the poll on Truth Social and on X on 19 September 2026. The Hill and TechCrunch both covered it. A day later he dropped Supreme Intelligence, “probably because of its relationship to the Supreme Court”, and restarted the vote with the other two. ↩
Sharkey said this in the BBC obituary of McCarthy in 2011. McCarthy himself was less sure, and told CNET in 2006 that he could not have used “computational” in 1955, because the idea that computers would be the main vehicle for the work was a minority view at the time. ↩
The Wikipedia article on the AI winter lists the alternative names and notes that they “helped to procure funding by avoiding the stigma of false promises”. Ginni Rometty explained the Watson choice to Bloomberg Businessweek: “The world calls it AI. There’s so much fearmongering about AI.” The quote is in Stephen DeAngelis’s piece on cognitive computing. IBM went back to saying AI within a few years. ↩
The Machine Intelligence series of workshops and volumes ran for decades under Michie as editor. The University of Edinburgh’s history of its early AI years covers the department. ↩
I first sketched this deal on LinkedIn. ↩

AI Decisions Just Got 100 Times Cheaper

Wed, 16 Sep 2026 16:54:32 +0100

A new kind of model is out. It makes decisions instead of writing text, it answers in well under a second, and it costs a hundred times less than the models you are used to. I tried it on everything I am building, and the results are at the bottom of this post. First, some background on why it matters.

Prompt engineering has always been rubbish. I wrote early last year about how it is the punch card era of programming all over again. If AI was so good at writing its own text, why could it not write the prompts as well?

Early in my use of large language models I came across DSPy, which solved a lot of the problems around my laziness with prompts.¹ It felt like an important step forward, and it felt like there was a useful product to be built on that idea. So I built my own version, Kaijo. It gave deterministic outputs and improved its own prompts: you gave it feedback, and it wrote the improvement for you. It even used its own system to create prompts for itself, which was both elegant and hard to get working. I ran a waitlist, gained some traction, used it on a few projects, and it was undeniably useful. However, something was always slightly missing.

Months later I worked out what that was. Optimising the prompts is only half the job. The AI also has to work out the structure of how it approaches the work. If I make the early decisions about how to break the job down, I am micromanaging it, and it never gets to find a better way.

Loops, Then Programs

My first step was to replace that structure with a powerful model and a loop, and let it work out the approach for itself. Loops are much simpler to run than any orchestrator, which is why I cut Ralph down to one line with no setup. This was a great start, and the job was normally done to a good standard. But even with the most powerful models there was some variability in the process, and letting the model call tools endlessly to work out what it needed cost a lot of tokens.

The next step is to codify that process. Ask the AI to write a structured, repeatable program which itself calls AI to get certain jobs done. Along the way you give the AI feedback where the structure does not quite work, and build up a set of repeatable cases, or evals, which it uses to check it has not gone off the rails while it makes improvements. I wrote up my latest thinking on this last week, when I explained why prompt evals alone are useless.

This breaks the use of AI into smaller pieces inside a larger program, which is a great way to build in the right amount of reliability. The repeatable parts matter as much as the AI parts. As I put it when I argued that you should stop saving tokens and start writing scripts:

The pattern matching is deterministic, so it lives in the script, where it runs in milliseconds and costs nothing. The judgement is not deterministic, so it stays in the prompt.

The Double Dovetail, from Stop Saving Tokens. Start Writing Scripts.

That is where some of the other Kaijo properties would be really useful. The self improvement not so much, as you need an agent to consider the whole build, but the single call simplicity, the guarantee that the output was well structured enough for a program to understand it, and the way you could choose a fast model for the job. Often you want a fast routing model at the beginning of the process, for example, as I talked about when I walked through building one for a client in my Make Your AI Rollout Stick webinar.

A Model That Decides but Cannot Talk

This week we got a new type of model with exactly those properties. TypeSafe calls it a System One model, trained with a process it calls Reinforcement Learning for Calibrated Decisions, and the first one is called Jev.² It can choose between options, give a score against some criteria, or answer yes or no as a probability. It brings nuance to a decision you might previously have hard coded, or paid an expensive LLM to make. It cannot generate text, but it can decide when you need text, so you call a traditional LLM only when you need one. It makes lots of decisions in parallel, it is fast, and it is very cheap: $42 per billion input tokens, with output tokens free.³ That is cheaper than DeepSeek 4.1 Flash on uncached input, with a guarantee that a machine can read every answer, and it answers all your questions at once.⁴

The guarantee is about the shape of the answer, not its truth. Jev cannot return an option that is not on your list, but it can still pick the wrong one, and the calibrated probability tells you how often that happens across many decisions, not which ones were wrong.⁵

High-res version available to newsletter subscribers

A Barrage of Questions

This is very powerful for systems that mix AI with regular code. We can hit any piece of input with every question that might be relevant, like asking for a whole multiple choice survey to be completed, and then let code act on the results. TypeSafe’s smart home demo shows the pattern. The code asks every question up front: is this a command or a question, which room, which device, which action. It throws away the answers that do not apply, and only hands over to a text generating model when someone needs a sentence back.⁶

Jev is behind a waitlist for now, but the approach is not exclusive to TypeSafe. Qwen’s open reranker models on Hugging Face already answer a yes or no question with a probability read straight from the model rather than generated as text, and more will follow.⁷

I Tried It

In the text adventure I am building, a change in the world has to be checked against every subsystem that might care about it, and a decision model can work out which ones need updating and whether an LLM needs to run at all. For my email triage, I can ask a batch of questions about importance, urgency and safety, including who a message concerns, quickly and cheaply, and discard anything that is not relevant before an expensive model ever sees it. For batch jobs like the recipe importer or meal generator at Cherrypick, choosing between a few hundred ingredient types when reading a recipe, and deciding how complex it is to cook, is exactly this shape.

As expected, there are a ton of little places to apply this in the agent builds I am running at the moment, including the successor to my Telegram bots, which is more productionised and which I will talk more about soon. So I tried it. I got Fable to read both codebases, work out from the documentation where a decision model could stand in for a decision an LLM makes today, and run side by side tests against Claude Sonnet 5 and DeepSeek V4.1 Flash using the eval setup I already had in place. Across email triage, the game’s action interpreter, recipe classification, my link reading agent and ten public benchmarks, that added up to around sixteen thousand decisions. Here is what I found.

It is good at closed lists. Sorting ingredient lines into twenty supermarket categories, it agreed with the reference answer 99% of the time, just ahead of Sonnet. It picked the right country from capital, region, languages and currency alone at every size up to the full list of 240, and given a made up list of a thousand shop orders, each with a status, it answered “is this order delivered” for every one of them in a single call, in just over a second, without a single mistake. On the ten public benchmarks it matched DeepSeek Flash and Sonnet 5 within a point or two, and beat both on toxicity. I tried injecting instructions into the input, and it seemed safe. French, German and Japanese scored the same as English. The probabilities are honest. When it is unsure it says so, and when it is confident it is nearly always right. The LLMs claimed near certainty on almost everything, including a fifth of the emails they dismissed that I needed to see. On email triage itself it was about as good as DeepSeek Flash and Sonnet at deciding what needed my attention: it agreed with my current setup three times in four where they managed four in five, and where it disagreed, an independent judge sided with it more often than not. That is useful. It can take the clear two thirds of my inbox in under a second and hand the rest to a bigger model.

It needs tuning and the right questions to make it work. In the game, with one untuned prompt, it called every attack and every persuasion a risky roll, and it asked for clarification far more often than it should. I gave every model one prompt and no tuning round, so treat the game numbers as a first pass. Unsurprisingly, it cannot multiply: “is 17 times 23 more than 400” came out right three times in four, with a bias to no, so do computation in code. Its confidence does not fall when the input is nonsense. I gave it a weather report where a customer complaint should have been and asked whether the customer wanted a refund, a replacement or an explanation, and it picked explanation at 0.99. Again, this was my issue. There was no option for “this is not a complaint”, and it will not invent one, so give it every option it needs to make a good decision, including the one that says the question does not apply. And there are some hard limits: at most 255 options in a choice, ten levels on a score, and about 32,000 tokens of context.

It is still a model. It makes nuanced decisions, it gets some of them wrong, and better models will make better decisions whatever the speed and the price. The important point is that this is the first model of its type. There will be more of them, and more powerful ones, able to make these kinds of decisions far better than this one can. A fair comparison gives each model a couple of rounds of revision before you judge it, and I would expect the game results in particular to move.

On reliability it held at ten requests a second for five minutes with no errors and every answer under a second. At twenty a second it started to fall over, with one request in ten taking eight seconds or more and a handful failing outright. Fine for the jobs I have in mind, and not something I would put a queue of a million items behind today.

I am not going to deploy it just yet, because it is one service and it has been public for a couple of days. I am keeping a very close eye on the technology in general, and when more than one provider offers a model like this, especially somewhere like OpenRouter, I am all for it.

Update, 25 September: That did not take long! Within two days of the launch, six open clones of Jev had appeared, all built from public models because TypeSafe has not released its own.⁸ This is good news if you are buying rather than building, because a variety of companies producing these models is what keeps you independent of any one vendor. The clones show that the architecture is easy to reproduce, but none of them comes with evidence you can check. Treat every number, from TypeSafe or from a clone, as a claim until you have tested it on your own data.

Decisions Just Got Cheaper

AI engineers need to learn this in detail. Chief executives and their leadership teams need to know that simple AI decisions on large amounts of data potentially got hundreds of times cheaper overnight. That matters for any content filtering, triaging or routing your teams do, whether you thought it was too expensive to hand to AI, or whether people are currently doing it by pasting things into Claude.

Within six to twelve months, once more models are trained this way, this kind of decision model will be in every custom AI product, and you can bet today that the frontier labs are either scrambling to build their own or scrambling to buy TypeSafe.

DSPy is a framework from Stanford for programming language models rather than prompting them. You declare what each step should do, give it examples, and it writes and improves the prompts for you. ↩
TypeSafe’s launch post, Introducing System One Models and Jev, by founder Diogo Almeida, who worked on the research behind ChatGPT at OpenAI. Jev answers three kinds of question: a choice between up to 255 options you define, a score against a scale you describe, and a yes or no returned as a probability. Every answer comes with a calibrated confidence, so code can decide when to escalate to a person or to a bigger model. ↩
TypeSafe prices input at $0.042 per million tokens and charges nothing for output, with responses in 70 to 500 milliseconds. Its own comparisons use the average of Astra and Fable as the reference answer, which it admits favours those labs. Mike Taylor at Every ran an independent test: 777 judgements across 37 documents in under 0.7 seconds for about a quarter of a cent, and in a seeded check Jev caught six of seven planted defects where Fable caught all seven, at roughly 25 times the speed and a 580th of the cost. ↩
DeepSeek released V4.1 Flash on 10 September 2026 at $0.15 per million input tokens off-peak and $0.30 at peak, with output at $0.60 and $1.20. Gemini’s cheapest current Flash-Lite is $0.25 to $0.30 per million input tokens. Jev is $0.042 with output free. The one rate that undercuts Jev is DeepSeek’s cache-hit price of $0.003 per million, which only applies to a prompt prefix the API has seen recently. ↩
The Geek in Review has the clearest independent read on this. Its example: 10,000 documents at 95% confidence still means about 500 wrong answers nobody has flagged, and calibration degrades when the data drifts from what the model was trained on. Test the calibration on your own data before trusting it. ↩
The smart home assistant demo in TypeSafe’s documentation. The code asks every question at once rather than waiting for one answer before asking the next, splits a multi part request into atomic commands with an LLM, and delegates to a conversational model only for questions it cannot answer with a typed decision. ↩
The Qwen3 Reranker models judge whether a document meets a query and return the answer as the probability of “yes” read from the model’s own output layer, rather than as generated text. It is one question at a time and not calibrated the way TypeSafe claims, but the shape of the idea is the same. ↩
Latent Space’s AI News roundup, Here are 6 Clones of Jev in 2 days, lists Laya, DiffusionGemmaJev, Bespoke Nimble, SemIf (formerly OpenJev), Jevlike and Kev-0.5B, which is small enough to run on a MacBook. Bespoke Nimble, a fine-tune of Qwen3.5-9B, reports 90% on its own curated eval against 93% for Jev. Laya’s confidence scores are based on entropy and are not calibrated, and calibration is the property that makes a decision model useful as a router. ↩

(I Am No Longer) Coding With AI

Fri, 11 Sep 2026 08:00:00 +0100

Coding with AI feels like the wrong title for these articles now. I am not coding with AI any longer. The AI is doing the coding, and I am doing the direction.

This is a bigger change than getting faster at writing software. I spend my time talking about what I want, trying what comes back and helping the AI find better ways to judge its own work. For my own projects, I am not reading the code at all. I am making sure good code gets written and good decisions get made.

A Conversation With the Codebase

Fable High and Astra High thinking are now my default workhorses for code: Fable High in Claude Code, Astra High in Codex. It is expensive: I am running the top-end subscription plans for both. But switching between them and asking them to review each other’s work has been the biggest unlock I have experienced so far. I can keep building with one, then have the other investigate what it has done and challenge its decisions.

With these models, I need far fewer tokens and review steps to get the work finished. That saves me time and attention, which is valuable enough to justify the cost.

I can now keep a stream of consciousness running with the codebase. I talk about what I want to build, notice something else, change my mind about a detail and keep going. I can mention 5-6 things whilst the agent is still coordinating the first one. It can handle that now, turning that conversation into work without requiring me to finish specifying everything before anything happens.

I used to obsess about tracking the AI’s work: first in Beads, then in notes on disk, then in GitHub Issues. I would put a whole bunch of work into issues and send agents away to do it, then become the bottleneck when lots of slightly wrong results came back. I described that cycle in Software Factories Can Handle UX: a dozen half-right screens, another round of corrections, then another dozen things to check.

Now I can keep multiple streams of work going by telling Astra what I want. It makes its own notes. I have barely read them, because it remembers the things I wanted to recall without me managing how it does that. I do not ask my assistant how she keeps her notes, so why would I ask Astra?

The conversation stays with one coordinating agent. I use a delegation skill to send pieces of work to other agents in the background, so the main conversation stays available. I can talk through the next change whilst something else is being built. The coordinator brings that work back, reviews it and fits it into the project.

I ask Astra to make sure it is creating a test / eval harness around the work: the tools, context and checks that let it act and see the result. That can mean starting the application, opening it in a browser or creating a way to inspect what it stored. I want to see the result quickly, work out where the friction is and go round again. Astra builds what it needs to make that possible. Evaluations are tests: they are all part of the feedback loop Astra needs to tell whether its work is correct.

My sessions mostly run on a remote computer I can reach from my phone, with a vault of Markdown files alongside the code. I have not had to arrange a special handoff between Fable and Astra. They save notes in files, so the next agent can pick up where the work has got to without making me explain it all again.

A Tighter Loop for Design

I have a slightly different loop for design. I use Claude Design and talk to Fable about what I want to see. It changes the designs quickly, I try them, and we iterate. Sometimes I make tiny copy tweaks or move something myself. I can work out how I want the product to feel before the coding agent commits that decision to an implementation.

Once I like it, I hand the design to an agent and go away for a bit. When I come back, it’s normally finished and has a fairly perfect version ready for me to review. The prototype gives it the screens, the copy and the behaviour to aim at. I do not need to turn every visual choice into a paragraph in a ticket.

The gallery below is Claude Design’s output for a training application I was working on with another developer. I could work through the screens and copy in the prototype, then hand over a visual reference for the implementation.

Claude Design's app for the training application was used as a reference for the developer implementing it. He did a great job of this first time.

The gallery stays alongside the code, and we update it when the product changes. This used to be a real pain, but now is a super quick job for AI. I can then tweak these designs with confidence, and the agent can build them.

Finding Better Ways to Review

I am attempting to run 2-3 projects at once in this way, returning to them through the day. I have to intervene much less to keep the implementation moving when I am working in Astra. It just cracks on. Its agency is extraordinary: I can leave it with something substantial and it keeps moving through the work. (That is, until it hits a usage limit. I’m using banked resets with Codex about once per day right now, on a x20 max plan!) No wonder Tibo announced they have paused new subscriptions to the $200 Pro plan.¹

I am much more directly involved in reviewing the result, and in giving the AI ways to review its own work. I also ask it to come up with creative ways for me to judge what it is building. I do not need to design every one of those tools myself.

One example is the text adventure RPG I have been building. You type what your character does, and the game tells the story through a traveller’s diary, with illustrations and a map of the world. The map has to fit the world the game describes. A convincing picture is not enough if its geography contradicts the story.

Astra built me a review form with the whole map and enlarged regions. It let me describe what I could see, then reveal the required geography and the earlier AI judgements. That gave me a way to help it work out whether it was judging another AI correctly. I could point to a particular mismatch and draw on my game-development experience without inspecting the code that generated the map.

Astra made the form so I could help it judge whether the map fitted the world.

I wrote about that in Prompt Evals Alone Are Useless. What struck me was how much of the review process the AI had built for me. I was giving it direction about what good meant, and it was finding ways to make that judgement possible. I am editing this blog post in the same way: filling in a form with comments for the AI on each paragraph.

The form I used to review the post you are reading.

Ensuring Good Code Gets Written

Years ago, I remember Jeffrey Fredrick telling me that my job as a CTO was not to make good decisions, but to ensure that good decisions were made. That is how I now feel about coding. My job is not to write good code. It is to ensure that good code is written.

For my own projects, that does not require me to read or see the code. I ask the AI to create a framework where it uses good engineering practices: refactoring as it works, keeping the codebase from growing unnecessarily and testing whether the overall system behaves properly.

A massively underrated part of coding now is getting the AI to make it easy for you to know whether what it has done is correct. That is work I can ask it to do, just as I ask it to build a feature. The map review form gives me something I can judge without first having to reconstruct the investigation myself.

Tests are part of that codebase too. I ask the agents to find duplicate tests and explain what failure each new test catches. A larger test suite can make every build slower without telling us anything useful. The same is true of instructions: I want the current decisions stated clearly, rather than a growing history of contradictory rules that the next agent has to untangle.

In July, testing my Slack agents from a phone kept turning me into the person who had to check every interaction. So I asked for a fake Slack that the coding agent could use to exercise the real application and inspect the replies. It cut out a lot of the boring early feedback I would otherwise have had to give: things that were obvious as soon as I tried to use it. By the time I reviewed the result in the real Slack client, my attention could go into more targeted, higher-level feedback.

Astra and Fable are now very good at using a browser, checking the interface and finding things that make it awkward to use. They can inspect screenshots, follow a journey and make corrections before I try it. I still come back and use the product, but the agent can do a great deal of the investigation first. Sometimes I just ask: “Please use the screens. Minimize the number of clicks I need to get between sessions. Come up with creative ways to improve this.” I might ask for a design mockup first, or ask it to change the flow on the page. Then I review what it has done.

Once the AI has built that environment, I can let it run. I want the system to contain the engineering judgement that helps anyone write good code, and to give the agents enough feedback to apply it.

AI Acting On Feedback Without Me

I am beginning to run AI automatically in response to user feedback. When a request or complaint comes in, an agent picks it up, checks it and works out what needs attention. It can investigate the problem and suggest improvements before I look at it. In my own applications, a background agent checks for that feedback on a timer.

That work comes back to my coordinating AI. It can show me what the feedback means, what it found and what it proposes to change. If I like the proposed changes, the coordinator can have them implemented. I do not have to start by reading the original request and working out what it means myself. The checks need to cover the whole system, including what the user sees, so the agent can tell whether a change actually improves the experience.

I have one conversation about what we are building, with investigation and implementation happening in the background. I use the system, come back, try it a bit, tweak the copy or ask for a change in behaviour, and repeat.

In The Harness Is the Bottleneck, I explored how much cheaper workers could do, and found that their bills left out expensive review and repair. Right now I am spending more on the models that let this whole conversation move with less intervention. I care about the cost of the finished work and the attention it needs from me, rather than the price of generating the code alone.

I am involved in what gets built, how we decide whether it is good and what happens when it falls short. The coding happens around that conversation.

I am no longer “Coding With AI”. I am no longer a coder at all. I am a director of systems, and a delegator of creation. I am moving higher and higher up the stack, and I’m not sure where this ends.

Tibo Sottiaux’s announcement, 10 September 2026. The pause applies to new $200 Pro subscriptions. Existing accounts are unaffected; other plans and the API remain available. ↩

AI Progress Is Accelerating. Here Is Why It Feels Slow

Fri, 11 Sep 2026 08:00:00 +0100

AI systems can finish longer tasks and recover from more mistakes. Falling model prices make more work worth trying. Those gains are easy to miss if you judge progress by asking a chatbot the same questions. The clearer test is what it can now finish that it could not finish before.

Progress in technical tasks is compounding. That does not guarantee a saving for your team: a better model can still create more review work than it removes.

Why It Feels Slow

A fluent answer quickly becomes ordinary. Expectations rise, and the remaining failures stand out. An agent can solve a difficult bug, then forget a requirement that matters more to you than its benchmark score.

Keep examples of those failures. As I argue in How to React to a New Frontier Model, testing familiar work and previous failures tells you more than trying to feel impressed by a launch.

Longer Task Horizons

METR measures the difficulty of a task by how long a skilled human takes to complete it. The 50% task horizon is the task length at which the AI system is predicted to succeed half the time. It measures human work time, not how long the AI runs.¹

Several 2026 systems reach tasks that take humans hours, up from minutes in earlier results. The trend is roughly exponential: each doubling adds more hours than the last. That is the acceleration I mean, even if the doubling rate stays constant.²

METR tests a model with its harness: the tools and software that manage its actions. The tasks are bounded technical work with clear success criteria. The human comparison has little prior context, like a new contractor, rather than a colleague who knows your systems.³

Succeeding half the time is too weak for most unattended work, and METR says estimates above 16 hours are unreliable. The chart marks that boundary. Use the trend as a reason to test longer tasks at the reliability your work requires, not as proof that AI can replace a working day.³

Better Tools, Checks and Recovery

An agent that can run a test, inspect a failure and repair it has a way to improve its own work. The harness gives it that feedback. Access to the right files, a working application and clear checks can change what the same model finishes.

In The Harness Is the Bottleneck, I describe how stopped runs and missing context distorted my coding experiments. A broken setup can look like a weak model.

Checks also need to reach the result a person sees. My text adventure game passed its mechanical tests whilst leaking internal bookkeeping into the story. Prompt Evals Alone Are Useless explains how I now test the model, code and what the game remembers together. A good answer in one call does not prove that the application works.

Cheaper Successful Work

The price of reaching a given benchmark score fell sharply in recent years.⁴ That makes more uses worth trying, but the bill per answer can mislead. Cheap attempts can leave expensive repairs for a person.

In my August coding comparison, the cheaper model’s bill excluded the stronger model that reviewed and fixed its work. Add the costs of all attempts, tools, review, repair, setup and maintenance, then divide by accepted completions.

Lower prices give us more room to check and retry. Better checks can reduce the repairs people must make. The saving is real only when the whole process delivers acceptable work for less.

Has AI Peaked?

The evidence does not support that claim. Technical task horizons continue to grow, and better harnesses make more of that capability useful. Neither trend guarantees that every new model will improve your workflow, or that progress will continue at the same rate.

If a pilot fails at the quality you need, identify the cause before investing further. Keep the failed examples and retest when a model or tool change addresses them.

Three Measures for Leaders

Choose a recurring task with a clear standard, such as drafting a monthly supplier report from approved records. Define success before testing: figures match their sources, required sections are present and claims have support. Compare similar work against the current process.

Successful Task Horizon

Track the longest type of task the system completes at your required success rate. Record attempted and accepted tasks, the model and harness, and any human help. Use human completion time as a rough measure of task size.

Cost Per Successful Completion

Divide the full cost of all attempts by accepted results. Hold the quality standard steady. Separate extra capacity from cash savings: time saved only reduces spending if expenditure falls.

Human Review Required

Record time spent checking, correcting and redirecting, including failed attempts. Separate routine approval from substantial repair. You have made progress when the same work meets the same quality standard with less money or human effort.

METR, Measuring AI Ability to Complete Long Software Tasks. Defines the 50% task horizon and analyses its growth. ↩
METR’s Time Horizon 1.1 data, checked 11 September 2026. The dashboard was last updated on 8 May 2026. The supplied chart shows the 50% horizon on a logarithmic scale and marks estimates above 16 hours as unreliable. The January 2026 update explains how expanding the task suite changes the fitted trend. Neither the fit nor the chart is a timetable for job automation. ↩
METR, Task-Completion Time Horizons of Frontier AI Models, checked 11 September 2026. The methodology explains the model-and-harness setup, low-context human baseline and domain limits. Estimates above 16 hours are unreliable with the current task suite. ↩ ↩²
Epoch AI, LLM inference prices have fallen rapidly but unequally across tasks, 12 March 2025. This measures prices at fixed benchmark scores, not the full cost of completed business tasks. ↩

Webinar: Make Your AI Rollout Stick

Thu, 10 Sep 2026 14:30:00 +0100

On 10 September 2026, I gave a webinar about the part of an AI rollout that never gets budgeted. The licences land, some training happens, and everyone builds their own version of their own work. Six months later the demos are impressive and the business runs at the same speed. Who owns the tools people have already built, and who fixes them when the AI changes or the person who made them leaves? If the answer is nobody, the rollout has not stuck yet.

Let people try things

A support worker builds a tool to spot repeat requests and tells nobody. Finance uses AI to read invoices and saves a few minutes a week. These trials are how people learn what AI can do for their work, but colleagues often know nothing about them, so teams end up solving the same problem several times.

Peter Seibel called this letting a thousand flowers bloom at Twitter.¹ The next step was to choose which tools to support. Trying things is a useful start, but a shared tool needs care beyond its first version. A Thousand Flowers Is Not Enough explores that next step.

Finish the work

Little’s law says the time to finish anything equals the unfinished work divided by the rate work gets finished. AI has raised the unfinished work. A pile of prototypes, meaning early working versions only one person can use, does not change what the business gets done. It adds noise: more half-built things to keep track of, and more for whoever has to check them. The business only speeds up when work leaves the building as a result people can use and trust.

A tool can also seem to work while hiding a serious fault. One executive built a tool in Claude to view client work. When they asked to use it from home, the AI put the data behind a public link. That exposed the company’s data, even though the tool appeared to do its job.

Where work waits

Writing code faster does not help if it then waits longer for someone to check it. A study of ten thousand developers found that heavy AI use produced 98% more pieces of work for review.² Reviews took 91% longer, while the pace of delivery stayed the same. Feedback Is the New Bottleneck brings together the sources on these delays.

Those figures describe software teams, but the same problem can arise elsewhere. A finance worker can use AI to compare a hundred sets of accounts, but someone still has to check them. The slowest step has moved from doing the work to reviewing it.

Tools need support

Codex, ChatGPT and Claude are good at helping one person on one machine get work done, and they are still rough at the edges themselves. Changing a whole process takes someone who understands its steps and can connect the tools it uses. Buying licences does not assign that job to anyone.

AI companies have committed nearly $10bn to engineers who work inside their customers’ businesses.³ The name for the role is forward deployed engineer, or FDE, and you will hear it a lot this year. They map how the work happens, connect the AI to your systems and data, decide what it can see and when a person reviews its work, and stay until it runs on its own. The labs send them to customers with thousands of staff, so everyone else has to find their own.

Who does the work?

At a few hundred staff, a small team of engineers you already have can start part time. At around fifty, it is one engineer, or a couple, with a clear job to do and a name against it. If you are on your own, the same discipline applies: decide what starts the tool, who checks it, and what it can reach before you build. The people who suit this work like mess. They will sit with finance for an afternoon or chase a missing login for a week, and that matters more than being your strongest software designer.

The people doing the work are best placed to choose what to improve, so point the engineers at the tools people have already built. Their job is to sit with the person who made it, understand what they wanted, and make it safe for others to use. Start with real data and aim for a working process in five days. Long rollouts stall while teams wait on each other.

Is it worth maintaining?

XKCD’s chart asks how much time a task can save over five years.⁴ Saving one minute a day allows four hours to build the tool. Saving an hour a week allows ten days. But the chart leaves out the time spent keeping the tool working.

AI can cut the time it takes to build something, but the systems it connects to will still change. The AI itself will change, and the person who built the tool will leave. Count the work of keeping it running when deciding whether it saves time.

One real process

One finance team needed to check sales orders against customer purchase orders. Each sales order arrived in the finance system, but someone still had to find the matching purchase order. The purchase order arrived as a PDF or email, sometimes in another language. Someone compared the amounts, codes and terms, then chased anything that did not match. This took one or two people several hours a week.

This is work I did recently with the team. We did not want it to start life as one person’s private tool that nobody else could use or knew about, so we mapped the process first, decided how it would run and who would review it, and then built it.

The shared version starts when a new sales order arrives. It can read the finance system but cannot change it. It records what it checked and any details that did not match. A finance person gets an email alert to review those differences. A weekly email lists what matched, using the format the team found easiest.

Keep ownership

When outside engineers do this work, whether a vendor, a consultancy or me, the knowledge of how your work happens can end up in their team, and that is how help becomes lock-in. Before anyone starts, ask four questions. Who owns the process maps? Who owns the checks and the records? Can we run it without them? Are they teaching our people, or becoming our AI department? Escape the Great AI Lock-In covers this risk in more depth.

Questions from attendees

One attendee asked how to let people try AI while keeping data safe. My answer was to use company accounts and let tools read data without changing it. Turn on features as people learn to use them, and explain which tools are allowed. Banning AI can push people towards personal accounts.

They also asked about a single view of all the AI processes staff had built. I did not know of a platform that offered one. Usage counts do not explain what people are doing. Seeing the content of their work also raises questions about privacy. Talk to frequent users to find useful ideas, and to those who avoid AI to understand why.

Another attendee asked how to test AI when its answers can vary. Use evals, which are checks that show whether the AI still does its job after changes. The checks must cover the whole process, as why prompt evals alone are useless explains. The code and instructions around the AI are the subject of The Harness Is The Bottleneck.

Key takeaway to remember

Name who owns each AI process, and give engineers time to turn the useful early versions into tools the whole team can rely on. Letting everyone try things is the right start, but it is not a rollout until somebody owns the result and the work of keeping it running is in the plan.

One thing to try this week

Draw the steps of a process that takes a day a week. Note what starts it, where work waits, and who checks the result. Mark who fixes it when it breaks. Find any version someone already built, then ask whether the time saved covers its upkeep.

Peter Seibel, Let a 1,000 flowers bloom. Then rip 999 of them out by the roots., written while he led the Engineering Effectiveness group at Twitter. ↩
Faros.ai study of 10,000+ developers across 1,255 teams. Reported in Ankit Jain, How to Kill the Code Review, Latent Space, March 2026. ↩
Tomasz Tunguz, The $10B FDE Boom. ↩
Randall Munroe, xkcd 1205, Is It Worth the Time?, CC BY-NC 2.5. The maintenance column is my addition. ↩

Prompt Evals Alone Are Useless

Wed, 09 Sep 2026 08:00:00 +0100

Over the last couple of nights I have been building a text adventure RPG, mostly to see what Astra could do. I have been giving the AI an end of day prompt, leaving it to run for hours, and seeing what it has done in the morning.

Playing the game is simple. It is told through a diary style interface, almost like reading an ebook: you type your character’s actions, you roll some dice if needed, and the AI narrates what happens with little illustrations, keeping notes and weaving a story as it goes. I will write more about the game itself soon. This post is about how I am building an eval system to improve it.

At the moment, the game is fun but inconsistent. The diary drifted out of tense, a safe action was treated as if it were risky, and the town in the prologue felt deserted. I noticed all of them as a player, and my instant thought for fixing them came from Kaijo, the AI email product I built to fix unreliable AI, and from the eval framework I described when I wrote up how to build a robust LLM application: save the example that went wrong, have an AI judge score it, rewrite the prompt, and try again.

It has grown into much more than that.

Astra built the prompt evaluation layer first. If I want to complain about what happened, a little flag button and a dialog let me save the complaint with the exact request we sent to the model. Astra can then pick it up in development, and a replay command sends that same request again with a changed prompt so I can compare the two answers. The agents rebuilt my three complaints as the first saved cases.

Then Astra started fixing things. The interesting part was that it was not just fixing the prompts. Almost every fix changed the shape the model has to fill in when it answers, the code around it, or the record of the story so far, and only a few lines of instruction changed with them. A prompt eval, a test that checks a prompt on its own, cannot tell me whether this game works, and neither can the ordinary tests AI writes for the code. I need to stress the whole system at once.

The Faults Were Not in the Prompts

The first fix came from a storm. The rules stopped my character setting out in foul weather, which was right, but the story then said “The attempt consumed the phase”, which is the game’s internal bookkeeping leaking into the prose. Every mechanical check passed, and Astra still rejected the run because the story was wrong. Astra changed the required answer format so that every story paragraph must name the game event it describes.¹

The second fault was a loose end in the story. Early on, a character called Mara lent me a cup, and the game is supposed to remember that I owe it back until I return it. If my character carried the cup around or checked their kit, the game counted that as progress towards returning it, even though nothing had happened. The fix was one sentence beside the answer format: “Continued possession, routine use, cleaning, checking equipment and reminders of a promise are not progress.” To place that sentence, Astra had to trace how the game recorded progress on the debt. That needed a whole new subsystem, with ordinary tests of its own, and also an end to end check to make sure the whole thing worked.

On an ordinary turn the game sees only the last three passages of the story, and everything older lives in a ledger, a running record the code keeps so that loose ends like the cup survive.² The code also rolls the dice, and a separate check looks at every picture before the player sees it. What the player reads is never the prompt on its own. It is the prompt, the answer format, the dice, the ledger and the picture check all working together, and a replay of one saved request can never tell me whether they did. All of that surrounding software is the harness. If the harness decides whether work lands, the eval has to test the harness. So I stopped treating evals as rows of data and started writing them as code.

All My Evals Are Code

This feels to me like an extension of Behaviour Driven Development, which I wrote about at length in earlier days, driving the whole system from the outside with end to end scenarios, only here the thing being checked is far more subjective and nuanced.³ So Astra and I drew on this and started putting a new system together.

I organise my tests and my evals into one pyramid. Ordinary software tests sit underneath, covering the game rules, the dice and the saving of progress, so the evals above them only ever have to question the model and the harness around it. The replay cases, which only check prompts, sit in the middle. Each saves the exact request and the model’s answer, and the replay command sends that request to the model again, refusing to run if the game has changed since the case was saved.

This infographic shows all of this at a glance:

High-res version available to newsletter subscribers

At the top sit scenarios written as code that test the whole game. Each one specifies the action the player types, a sentence saying what should happen, the starting conditions, the dice results to fix in advance, and the checks to run afterwards. Each scenario runs the game, calls the model and saves progress into a temporary copy of the database, with pictures switched off because the picture check has its own evals. I fix the dice because the model’s answers already vary from one run to the next, and adding the game’s own randomness on top would make a run impossible to judge. The blind travel case looks like this, with one check shown:

{
  action: "Lost in thick mist, I trek east from Whitebank toward the mapped land there, accepting that I may drift off course.",
  purpose: "The interpreter declares an eastward Trek and explicit blind navigation; held die 2 resolves the left-hand actual edge, whose encounter state drives the continuation.",
  validateInterpretation(interpretation) {
    assert(interpretation.travel?.navigation?.lostOrMist === true, "The explicit mist was not classified as blind navigation");
  },
}

The purpose sentence tells whoever reads the result what to expect: the player heads east, accepts they might get lost, and drifts the way the fixed die says. The checks then inspect what the game recorded, and a report keeps those results beside the story the model wrote, so the reason for running the case survives with the output. Even a case with every mechanical check passing still needs someone to read the story.⁴

The Agent Is the Judge

The combat report leaves the story column marked “Not graded; coordinator inspection is required.” and hands it to the coordinator, which is Astra in Codex or Fable in Claude Code. The coordinator compares the story and the recorded results with the expected outcome, then traces any mismatch through the saved game, the answer format and the code to find where the fix belongs. The cause can be an event several turns earlier, which is why the judge also needs a longer session.

This morning I needed to know whether the AI checking the game’s maps was judging them properly. Astra made me a review form with the whole map and enlarged regions to inspect. It asked me to describe what I could see before revealing the geography the game required, then kept the earlier AI judgements hidden until I had made my own assessment.

I found this extraordinary. Astra had made me a form to help it work out whether it was judging another AI correctly. The AI spotted the problems but did not know how to fix them, so I gave it some pointers from my game-development experience. I have never worked in such a light-touch, high-level way before.

The review form Astra built, with my notes from inspecting the map.

The Long Session Is the Real Test

The model’s memory is trimmed as a story grows, and that trimming can lose the details that make a story worth following long before any single reply looks broken. One six turn scenario tests whether the ledger keeps the cup debt alive after the loan drops out of the three passages the model can see. The game must remember the debt until the cup is returned, and that test is what exposed routine actions being counted as progress.

A separate Day 120 case checks whether the game still remembers unfinished business from day one when the three recent passages have nothing to do with it. Once the agent can judge a whole session, the screen is the last part of the player’s experience left outside the loop.

Right Through to the Screen Itself

The agent drives the browser and takes screenshots of the whole user experience as it works, noting page errors, failed requests, the text on screen and anything that spills outside its box. Reading its own screenshots lets it inspect what the player sees, so there is no separate automated screen test. With Fable and Astra, this loop has finally closed for me. Once the agent has had a good go at the screens, I try them, and they mostly work first time with a few tweaks.

This builds on how I use galleries to give UX tickets a visual specification, which gives the agent something concrete to aim at and inspect. My feedback now leads through a code change, a scenario, a session and a browser check before I try the result again, so the eval covers the whole path from a complaint beside Go to the screen where I can see whether the fix helped.

What Evals Are Becoming

The system improving this game now changes code, tools and answer formats, along with memory and screens, guided by failures from whole runs. Research projects such as Self-Harness and Weco’s AIDE2 test the same idea, one agent improving another by changing the software it uses.⁵ Andrej Karpathy’s autoresearch runs a similar loop against a single number, and Shopify has generalised it into pi-autoresearch, which will chase any metric you can measure.⁶ This game has no such number. Whether a story is good is a judgement, so the judge has to be an agent reading the whole run. The whole eval setup came together in about two days, with me directing Fable and Astra and their delegated workers doing the building.

A system that improves itself needs guardrails, and it needs to run in a sandbox. Otherwise, as the comic shows, you never know what your agent might do while you are out of the room. Every scenario here runs against a temporary copy of the database, and since the second evening every agent reruns the saved scenarios and reads the results before handing work back. The cases live with the game code, where the agents can read and change the tests too.

For your own product, take a complaint from using it and write down what a run that reproduces it should show. Give that run enough of the real system to reach the fault, preserve its failures, and let the coding agent read the output alongside the code. Keep replay cases where a single call answers the question, then extend the scenarios through lost memory and onto the screen. I will keep playing the game and pressing the flag when something feels wrong, giving the agents another concrete example of what good should look like.

The storm fix had three parts: every story paragraph must name the game event it describes, the model provider enforces that requirement through the answer format, and one prompt instruction bans the bookkeeping phrase. Astra then reran the earlier scenarios and read the results to make sure nothing that used to work had broken, and the three failed attempts stay on record beside the passing one so the history is not lost. ↩
On an ordinary turn the model also sees a list of established facts and a repeatable selection of unfinished storylines, including ones that have not appeared recently. The ledger holds a record for each thread as it is set up, progresses, pays off and closes, with a short summary and a quote copied by code from the passage it came from. ↩
Dan North’s original introduction to BDD describes writing each test as a sentence about behaviour, driven from the outside of the system in. Every scenario in this post follows that shape, with a model in the loop. ↩
Evals like these need habits that ordinary tests do not. If the model provider fails mid run, the run says nothing about quality and must not be counted as a pass. Failed attempts stay on record, because rerunning until one passes would hide the behaviour I am trying to see. Changes to saved cases are recorded, and tests of the whole game need updating more often than prompt tests because they depend on more of the software. ↩
Self-Harness, by Hangfan Zhang et al., describes agents mining failure traces, proposing minimal harness edits and accepting them only after regression tests. Weco’s AIDE2 write-up describes an outer loop rewriting the inner agent’s harness, including its search policy, context compression, an evaluation bug fix and reward hack reduction. Both sets of results are self reported by their authors. ↩
Andrej Karpathy’s autoresearch gives an agent a small training setup, a five minute budget per experiment and one metric, then leaves it overnight to keep the changes that improve the score and discard the rest. Shopify’s engineering team generalised the loop into pi-autoresearch, an extension for the pi coding agent that works on any measurable target, and reports using it on more than forty metrics across the company. ↩