My selftest said 3/3. One of the three had never fired.

I spent a day putting a two-month-old release out of its misery — the tag had been sitting there since June with zero assets, which meant the only way to run any of my OCR tooling was to clone it and build it. There was no door. That is not a marketing problem, it is a door problem.

Four separate measurements lied to me before the day was out. I want to write them down together, because in isolation each one looks like carelessness and together they look like a pattern: **every one of them lied in the shape of a result.** None of them errored. Each produced a number, and the number was wrong in a way that agreed with something I already believed.

One: the build gate that failed a healthy build

I wrote a release script that refuses to ship a binary which cannot print its own usage. Reasonable. This was the check:

set -euo pipefail
if ! "$OUT/$product" 2>&1 | grep -q "Usage: $product"; then
  echo "✗ refusing to ship it" >&2
  exit 1
fi

It fired immediately, on a binary that was completely fine.

Both my CLI tools exit non-zero when run with no arguments — that *is* their contract, and it is asserted elsewhere. Under `pipefail` a pipeline reports the rightmost non-zero status, so the pipeline returned 1 even though `grep` matched perfectly. The gate was reading the binary's exit code while believing it was reading grep's verdict.

The fix is to stop piping and hold the output:

usage="$("$OUT/$product" 2>&1 || true)"
if [[ "$usage" != *"Usage: $product"* ]]; then …

I have known "don't check `$?` through a pipe" for years. I still wrote it, in a brand-new file, in a gate whose entire purpose was rigour.

Two: the flag that did not exist

I wrote the release notes before I wrote the download. The first line of the install instructions was:

./scan-ocr --help

There is no `--help`. The tool treats `--help` as a filename and reports that it cannot open a PDF at that path. The very first command a stranger would type on the very first page they would read was wrong.

Nothing caught this except unpacking the actual tarball, in a scratch directory, and typing the line. It could not have been caught by reading, because I wrote the line from what I assumed the tool did — and I wrote the tool.

Three: the selftest that passed for the wrong reason

This is the one worth the post.

I have a discipline that a check is not evidence until you have watched it fail. So every gate I write carries a `--selftest` that deliberately breaks the rule and requires the gate to notice. It printed:

✓ a fixture with no frontmatter — caught
✓ a fixture with no reading direction — caught
✓ a fixture whose headings are inside a fence — caught

3/3 rules proven to fire.

Three out of three. The check checks. Move on.

Then I fixed an unrelated problem — one fixture in the corpus was not declared in the manifest — and the selftest dropped to **2/3**.

Here is what had been happening. The selftest builds a corrupted copy of the corpus and asserts the checker's failure count goes up. The undeclared fixture made the checker fail *on every run*, corrupted or not. So each case saw the count rise, recorded "caught", and moved on. The corruption was never what it was detecting.

With the corpus clean, the truth came out: the fenced-heading case had never fired. It fenced one heading in a fixture that had two chapters, so the other heading still stood and "this file has at least one chapter" kept passing. It now corrupts a single-chapter fixture, and 3/3 means it.

**A selftest can pass for the wrong reason, and it looks exactly like passing for the right one.** The failure mode is specific and I had not seen it before: an ambient failure elsewhere in the system will happily impersonate your probe's signal, and the greener your suite, the more likely this is invisible. The thing that exposed it was not scepticism. It was cleaning up something unrelated and watching a number I trusted get worse.

Four: a rendering catastrophe that was entirely my ruler

The last piece of the day was measuring whether WebKit lays out a typeset book cover the same way Chromium does — the browser is the only reason a headless Chromium is in the dependency list at all, so if the answer is yes, that dependency can go.

I rendered the same HTML in both, diffed them, and got:

title line 1   WebKit w=-1600 ink=0 | Chromium w=1022 ink=54756  → Δink -100.00%
title line 2   WebKit w=-1600 ink=0 | Chromium w=842  ink=40987  → Δink -100.00%

Zero ink. Every line. That is not a rendering difference, that is a renderer producing a blank page — and for a moment I believed it, because "WebKit can't do this" was a story I was half expecting.

WebKit snapshots at the display's backing scale. I asked for 1600×2260 and got 3200×4520. I was reading a 3200-wide buffer with 1600-wide indexing, so every row I sampled was the left half of a different row. `-1600` as a width should have stopped me on sight; a negative width is not a measurement, it is a `min` and a `max` that never met.

Once both images were normalized, the actual answer was the opposite of the panic:

specimen	differing pixels	ink coverage
Latin, two-line display title	0.48%	2.74% vs 2.74%
CJK, vertical, right-to-left	0.27%	**ink bounding box identical in both dimensions**

The vertical CJK case was the one I expected to break — strict line breaking, no word gaps to balance on — and its ink box matches to the pixel. The remaining difference sits in one-to-fifteen-row bands at the cap-height and baseline edge of each line, with 0.0–0.4% through the middle of the same line. That is glyph hinting, not layout.

Worth noting what I did in between: I blurred both images to test whether the difference was antialiasing, and blurring made it **worse**, which seemed to disprove the theory. That test was run on the mismatched buffers too. A wrong ruler does not just give you a wrong number — it gives you wrong answers to the follow-up questions you ask to check the first one.

What actually held

The pattern is not "be more careful." Three of these four were written by someone who knew the rule being broken.

What separates them is how each was caught:

1. **The pipefail gate** — caught because the gate ran, on a real binary, and disagreed with reality.
2. **The missing `--help`** — caught by unpacking the shipped artifact somewhere else and typing the line.
3. **The selftest** — caught by an unrelated cleanup that changed a number I had already accepted.
4. **The ruler** — caught by a negative width, and confirmed by checking the image dimensions I had never checked.

Not one was caught by reading code. Every one was caught by running something and looking at what came back, and in the two worst cases by looking at something I was not investigating.

So the rule I would actually hand someone:

**A measurement that confirms what you expected is the one to audit.** The three lies here that survived longest were the ones that agreed with me — WebKit can't do this, my gates are rigorous, my tools have `--help` like every other tool.

The corollary is the useful part. Validate the ruler before you blame the worker. When a metric goes red, the first hypothesis should be that the instrument is broken, because a broken instrument and broken code look identical from where you are standing — and the instrument is usually the thing nobody has tested.

The boring footnote

All of this came out of reepub, a Mac tool that turns documents you own into clean, reflowable EPUB3 using Apple's on-device Vision OCR. No API key, no account, no network call in the conversion path — MIT, and the source stays open because that claim is only worth anything if you can check it.

The release now has binaries in it, which was the entire point of the day. The OCR ships as its own command-line tool, in case the OCR is all you came for.