A review queue reported zero items waiting, which is the same string a healthy fully-reviewed queue prints. It was not healthy. Two correct components combined into a pipeline that had starved its own sampling pool to nothing. The fix restored the pool from 0 to 83, proven by a two-way set-equality differential, with the human-graded count correctly unmoved at 2.
Key facts. Sampling pool starved to 0, restored to 83. Human-graded count unmoved at 2. Gate needs 5.
Last updated:
01The queue was empty because the system ate it
For days, a number on my dashboard read 2 of 5, meaning two of the five required human reviews were done and I owed three. I planned evenings around it. Here is the answer up front, since it took an audit to find it: I was not behind. The review queue was impossible by construction. 83 of the 85 rows in the gold file were machine output, the sampler retired every task that carried one, and no component in the pipeline had any way to say so.
This is a story about the worst failure shape I know: the one that reports nothing. Not a crash, not an error. A clean, confident zero.
02Five graded examples before anything counts
My automation system's work is scored by a Judge, another piece of software that grades his output. I do not trust a grader I have never graded, so the rule is written down: the Judge's verdicts count for nothing until it agrees with a human on at least five real examples. Those human verdicts are called gold, and the requirement always travels with its measured state, which sat at 2 of a required 5.
Read cold, 2 of 5 is a to-do list with my name on it. That is how everyone read it, including the system's own status reports, which filed it under waiting on the human.
03Two correct components, one starved pipeline
The sampler is the tool that serves items for review. Its rule was sensible: if a task already carries any gold row, it has been judged, stop serving it. Meanwhile an analytics pass had been writing mechanical rederivations, machine-recomputed labels, into the same gold file. The tool's own help text says those rows are NOT human gold. The sampler does not read help text. It saw gold everywhere, retired everything, and the pool drained to zero.
Then it reported 0 unadjudicated. Which is true. It is also exactly what a healthy, fully reviewed pipeline prints. The failure existed only in the composition of two locally correct components, which is why no single one of them could report it. Actor-first accounting matters here: the analytics pass ate the pool, the sampler enforced its rule, and both were behaving exactly as written.
0483 of 85: machine work wearing a gold sticker
Only two rows in the entire gold file were verdicts a human had made. And it gets one notch worse. A second consumer, the report that decides when the Judge counts as calibrated, gated correctly on the human count. But the precision figure it would print the moment that count reached five was computed over all 85 pooled rows. The trigger was honest. The number behind it would have been mostly machine agreeing with itself, which is the pipeline's version of grading its own homework and initialing it.
05The second lock on the door
Suppose the pool had never starved and an item actually reached me. The function written specifically to recover the evidence behind each label, the files, the output, the thing you would actually judge, was wired into a different subcommand and nothing else. A served sample displayed a correlation id, a timestamp and the machine's own verdict, with no files, diffs or output behind it.
So the homework was impossible twice over. No items could be served, and a served item could not be graded. Two independent locks, and zero alarms between them.
06The fix, proven by differential
The fix was not accepted on vibes. Pre-fix sampling pool: 0. Post-fix: 83. The returning set is exactly the 83 mechanical rows, set equality checked in both directions, so the fix restored precisely what the bug consumed and nothing else. Just as important, the human gold count stayed at 2, because those two really are the only verdicts a human made. A fix that moved that number would have been a second bug wearing a cape.
The starved pool also became a standing regression gate, gold-pool-not-starved, and under house rules it had to prove it could fail before it was allowed to pass.
07What does an empty review queue actually mean?
The shape generalizes well past review queues. A backup job that writes nothing, a scanner that finds nothing, a queue that serves nothing: all of them print success. The only question separating a healthy zero from a dead pipeline is the denominator. Zero out of what, and who counted the what. I now ask it of every system that has been quiet for a while, and the quiet systems are most of them.
Nothing in this incident went wrong by any component's local definition, and the whole was still broken for days with my name on the blame line. Silent success is the worst failure shape there is. Make your quiet systems prove they can be loud.
08How every number here was derived
Both source files are append-only, so re-running these will give larger denominators than the ones quoted.
# Every number on this page, re-derived from the artefact rather than from a handover:
wc -l < ~/.sgnk/evals/reward-gold.jsonl
# -> 85
grep -c mechanical-rederivation ~/.sgnk/evals/reward-gold.jsonl
# -> 83
python3 ~/.sgnk/bin/reward-adjudicate.py --report | head -4
# -> n_human_gold 2, min_n 5, n_mechanical 83
cd ~/.sgnk && git show a4a932a^:bin/reward-adjudicate.py | sed -n '335p'
# -> done = {r.get("correlation_id") for r in gold_rows()} the pre-fix predicate
sed -n '435,436p' ~/.sgnk/bin/reward-adjudicate.py
# -> the same line today, filtered on HUMAN_SOURCES
cd ~/.sgnk && git show a4a932a^:evals/reward-gold.jsonl | jq -r .source | sort | uniq -c
# -> 2 incident-documented, 83 mechanical-rederivation, at the parent of the fix commit
jq -r 'select(.id=="gold-pool-not-starved")|.registered' ~/.sgnk/state/regression-gates.jsonl
# -> 2026-07-29T05:23:15Z
bash ~/.sgnk/gates/assert-gold-pool-not-starved.sh; echo $?
# -> 0