Never judge a headless AI run by its exit code

If you have started firing coding agents headless — `codex exec`, `claude -p`, a fanned-out batch of them — you have hit the question that has no obvious answer: **how do you know it worked?**

The instinct is thirty years old and completely wrong here. You check the exit code.

Exit 0 is not a claim about the work

An agent CLI that runs out of quota mid-task will tell you so in its output, write nothing to disk, and exit **0**. It did not crash. From the shell's point of view nothing went wrong, and the shell is right — the process ran, said its piece, and terminated normally. "I could not do the work" is a perfectly successful execution of a program.

So the first version of my headless runner did the obvious upgrade: stop reading the exit code, start checking whether the file the task was supposed to produce is there.

That is the right idea. It is also where the interesting failures start.

Trap one: yesterday's file

Run the same task twice. The first run works and leaves `out/report.md`. The second run fails — limit, timeout, bad prompt, doesn't matter — and writes nothing.

Your check asks "does `out/report.md` exist?" It does. Yesterday's does.

A failed run reported as a success, with a plausible, readable, *wrong* artifact sitting in the output directory to back it up. This is worse than a crash, because everything downstream now proceeds on stale data with no signal that anything happened.

The fix is to make presence insufficient: the artifact must appear **or its timestamp must change**. And a `--fresh` flag that deletes the artifact before the run, for when you want the strict version.

(Small portability note that cost me a debugging session: if you reach for `stat` to get that mtime, write the GNU-first form and fall back to BSD. A BSD-first implementation returns garbage on Linux rather than failing, which is the worst way for a portability bug to behave.)

Trap two: the file that exists and says nothing

Version three of the check: the artifact must exist, and be newer, and be non-empty.

I fanned a prompt out across several accounts in parallel and one leg came back a "success" with nothing in it. The generator had written no content at all, and the wrapper had done this:

printf '%s\n' "$result" > "$out"

With an empty `$result` that still writes one byte — the newline. One byte is non-empty. The check passed.

The lesson is narrower than "check for empty files" and more useful: **judge the thing you captured, not the container you put it in.** File size is a property of your own plumbing. Whether the model produced anything is a property of the output you actually hold in a variable. Test that, before it ever touches a filesystem that will helpfully add a byte for you.

Why the tests did not catch any of this

Every test prompt in the suite was a single line.

That is also how a fourth bug lived undetected: the recipes that framed a prompt for each engine were newline-delimited, so a multi-line prompt **shattered into one argv item per line**. Every real prompt I typed was a paragraph; every prompt the tests typed was `hello`. The framing is NUL-delimited now, but the point is not the fix. The point is that a fixture which never resembles real input is not a test, it is a ritual.

Two of these bugs — the shattered prompt and the one-byte success — were found by pointing an adversarial reviewer at the diff, on a different model family, with no memory of why any of the code looked reasonable. Both were invisible to me, because I knew what each line had meant to do.

The rules that held

If you are wiring up headless agent runs, these are the four that have survived contact with reality here:

1. **Judge by artifact, never by exit code.** The process exiting cleanly tells you the process exited cleanly.
2. **Presence is not proof — require appearance or a changed mtime.** A previous run's output will happily impersonate this one's.
3. **Test the captured output, not the written file.** Your own `printf` will manufacture a byte of false evidence.
4. **Close stdin, and impose a timeout.** An agent CLI that decides to ask you something will otherwise wait forever. On stock macOS there is no `timeout` and no `gtimeout`, so the portable fallback is a `perl` alarm — one line, always present.

There is a fifth that is less a rule than an observation. I later received a set of orchestration scripts from a collaborator who had solved the same problem independently, and they had arrived at three of these four — judge by output, close stdin, `perl` alarm when there is no coreutils `timeout` — without ever having seen my code.

**Two parties converging on the same rule is the rule earning its keep.** It is the closest thing to proof you get in this corner of the craft, where nobody has been doing it long enough to have a canon.

The boring footnote

This lives in clikae, a small local layer for people running several AI coding accounts — plain bash, MIT, no daemon, no telemetry. Its headless runner is called `burn`, which is named after what it does to your quota, and after what it did to me before it learned to check.