A dashboard read 2 of 5 human reviews for days, so I assumed I was behind. I was not. An analytics job had written 83 machine rows into the file that holds human verdicts. The sampler retired every task carrying one, and the queue drained to zero. It then reported zero unadjudicated, which is exactly what a finished queue reports. Any queue whose empty state means both done and broken needs a separate liveness counter.
Key facts. 85 rows in the gold file, 83 written by a machine. Sampling pool: 0 before the fix, 83 after. Human count unmoved at 2.
Last updated:
01My review queue printed the same zero whether it was finished or dead
My AI coding system scores its own work with an automated grader. Nobody should trust one of those on faith, so I built the standard safeguard. A human grades real examples. You measure how often the grader agrees. Its scores count for nothing until that agreement holds over at least five examples.
For days a number on the dashboard read 2 of 5. Two human reviews done, three owed. I planned evenings around it.
I was not behind. The queue was impossible. An analytics job had been writing machine-computed labels into the same file that holds human verdicts. The sampler saw them, concluded every task had already been judged, and served nothing.
Then it reported zero unadjudicated. That is true. It is also exactly what a fully reviewed queue reports.
0283 of 85 rows were machine output wearing a human label
The file that holds human verdicts is called the gold file. It had 85 rows. Two were verdicts a human made. The other 83 were mechanical rederivations, machine-recomputed labels written by a separate analytics pass.
That tool's own help text says those rows are not human gold. The sampler does not read help text. It reads the file.
It gets one notch worse. A second consumer, the report that decides when the grader counts as calibrated, gated correctly on the human count. But the accuracy figure it would print the moment that count reached five was computed over all 85 pooled rows. The trigger was honest. The number behind it would have been mostly the machine agreeing with itself.
03Root cause: two correct components, one broken composition
The sampler's rule is sensible. If a task already carries a verdict, it has been judged, so stop serving it. Correct in isolation.
The analytics pass writes its rederivations to the gold file. Also correct in isolation; that is where labels live.
Neither component contains the bug. It lives in the join between them, which is why no single component could report it. The sampler enforced its rule exactly as written. The analytics job wrote exactly what it was built to write. Both behaved correctly and the pipeline was dead for days.
There was a second lock on the same door. I had written a function to recover the evidence behind each label, meaning the files and output you would need to judge it. Then I wired it into a different subcommand and nowhere else. So a served item would have shown an id, a timestamp and the machine's verdict, with nothing behind it. The homework was impossible twice: nothing could be served, and a served item could not be graded.
04The fix, and the differential that proved it
The fix is to filter the sampler on the source of each row rather than on its existence. One predicate.
I did not accept it on inspection. Sampling pool before the fix: 0. After: 83. Those 83 are exactly the machine rows, checked as a set in both directions, so the fix restored precisely what the bug had consumed and nothing else.
Just as important, the human count stayed at 2. A fix that moved that number would have been a second bug wearing a cape.
The starved pool then became a standing regression check, and under house rules it had to prove it could fail before it was allowed to pass. Note what it asserts: that the pool is not starved, which is a count of what can be served. It is not a reading of the sampler's own report of its health. A component cannot be the witness for its own liveness.
05How to check a quiet queue of your own
The generalisation is one sentence. Any queue whose empty state means both done and broken needs a separate liveness counter, not a status check.
The shape is everywhere once you look: a backup job that writes nothing, a scanner that finds nothing, a queue that serves nothing. All of them print success. Telling a healthy zero from a dead pipeline takes a denominator, and it has to be counted by something other than the component reporting the zero.
06What this buys, stated plainly
Nothing in this incident went wrong by any component's local definition, and the whole thing was broken for days with my name on the blame line.
Silent success is the worst failure shape there is. Monitoring it requires having predicted it. A crash announces itself; a clean, confident zero does not.
Judge validation before trust is standard practice, and Hamel Husain's guide describes the loop well. The safeguard is not the story. The story is that a correctly built safeguard sat on top of a starved queue and reported healthy numbers the entire time.
07How every number here was derived
These commands read live state, so re-running them will give different numbers from the ones quoted here, usually larger. That is the point of printing the commands rather than only the conclusions.
# Every number on this page, re-derived from the artefact rather than from a handover:
wc -l < ~/.sgnk/evals/reward-gold.jsonl
# -> 85
grep -c mechanical-rederivation ~/.sgnk/evals/reward-gold.jsonl
# -> 83
python3 ~/.sgnk/bin/reward-adjudicate.py --report | head -4
# -> n_human_gold 2, min_n 5, n_mechanical 83
# The pre-fix predicate: any gold row at all retires the task
cd ~/.sgnk && git show a4a932a^:bin/reward-adjudicate.py | sed -n '335p'
# -> done = {r.get("correlation_id") for r in gold_rows()}
# The same line today, filtered on human sources only
sed -n '435,436p' ~/.sgnk/bin/reward-adjudicate.py
# The differential: what the pool held before and after
python3 ~/.sgnk/bin/reward-adjudicate.py --sample --dry-run | wc -l