An audit agent reported leaked GitHub tokens at a named file path. The path did not exist and never had. Dismissing the alarm there would have been the mistake: widening from the path to the pattern found real token-shaped strings in the agent's own session transcripts, a location nobody had listed as a risk. A wrong address can still point at a real fire.
Key facts. Alarm named a path that never existed. Real vector: session transcripts. Pattern and denominator in the body.
Last updated:
01The audit was wrong about everything except the fire
An automated security audit told me there were leaked credentials in testing/workspaces.json. That file did not exist, and had never existed. There was still a leak: GitHub-token-shaped strings sitting in plaintext in my AI's own conversation transcripts, files nobody had named and no audit had been pointed at. The alarm had the wrong address and a real fire, and the fire was found only because the search was allowed to outgrow the pointer.
That is the whole story, and the whole lesson. An alarm from a machine is a hypothesis. You do not repeat it as fact before a live check, and you do not dismiss it on a wrong pointer either. Everything below is detail.
02A very specific alarm
The auditor was one of my automation system's subagents, dispatched to sweep a repository for security problems. It came back with a finding that had everything a serious finding has: a named path, a named credential type, and real stakes. Leaked personal access tokens in testing/workspaces.json. A personal access token is a password-shaped string that lets software act as you; on GitHub, whoever holds one is you, as far as the platform can tell.
This is the class of alarm you cannot sit on. It is also the class you most want to forward untouched, immediately, to look responsive. That is the trap. A specific claim feels pre-verified. Specificity is not evidence. It is just detail, and a language model produces detail at the same speed it produces truth.
03The empty room
House policy in this workspace is that no factual claim gets repeated before a live check, so I went to the file first. There is no testing/workspaces.json. Not moved, not renamed, not deleted in some recent commit. The path was invented.
Two comfortable exits open up at that moment. One: close the whole finding as a hallucination and move on. Two: soften the report to a possible leak and hand the ambiguity to someone else. Both are the same mistake in opposite directions, treating a subordinate's alarm as a verdict rather than a hypothesis. I nearly took the first exit. An alarm with a wrong address reads exactly like a false alarm. It is not the same thing.
04Where do leaked credentials actually end up?
So the question changed shape. Not, is the audit right about this file. Instead, are there token-shaped strings anywhere they should not be. That wider search came back positive in a place no audit had been pointed at: the AI's own session transcripts.
Every session my automation system runs is transcribed to disk, line by line, as JSONL under his project logs. Paste a secret into the chat, or let a tool echo one back, and it sits in those files in plaintext until someone goes looking. The auditor had, in a sense, smelled smoke correctly and then invented a room to put it in. The address was never the load-bearing part of the alarm. The credential type was.
I cannot interview this employee, and I cannot tell his right days from his wrong days by tone. His confidence carries no information either way. Which is why the workplace demotes every claim one level on arrival: alarms enter as hypotheses by policy.
05The rule it became
The incident is preserved as Learned Rule #2 in the standing rulebook the AI reads at the start of every session, in a rules block seeded 2026-06-27: verify a subagent's security or factual claim live before repeating it as fact, and attribute anything unverified.
The rule is deliberately symmetrical. It bans repeating the alarm unverified, and it bans dismissing it unverified. Because the rulebook is append-only and loads every session, the cost of the incident was paid once and the lesson compounds daily. An incident you survived is worth exactly what you wrote down.
06Standing orders for alarms
Four standing orders fell out of this, and they apply to any subordinate whose confidence you cannot calibrate. Never repeat an alarm as fact before a live check, however specific it sounds. Never dismiss it on a wrong pointer, because wrong address, real fire is a common shape. Widen from the named location to the underlying pattern. And write the outcome into a rule the worker rereads every session.
None of this is exotic. It is how you would manage a brilliant employee you were never able to interview. The difference is that a human auditor who invents a file path gets a hard conversation, and mine got a rulebook entry, because the alternative to him is me doing every audit alone. The fire in room four was real. There was just no room four.
07How every number here was derived
Both source files are append-only, so re-running these will give larger denominators than the ones quoted.
# The rule, verbatim, from the standing rulebook (verified this run)
grep -n -A 4 "Verify a subagent/audit" ~/.claude/CLAUDE.md
# The shape of the original live check: the claimed path returns nothing
find . -path "*testing/workspaces.json"
# The shape of the widened search that found the real leak vector
grep -rlE "gh[pousr]_[A-Za-z0-9]{20,}" ~/.claude/projects --include "*.jsonl"