My note from that night says sixteen nightly database backups were dead, and nothing reported it, because a job that does nothing and a job that works both print the same quiet output. macOS was refusing the scheduled task at a permission layer no log line mentioned. Silence is not a pass. Every quiet system should be made to say zero out of what.
Key facts. My note says 16 nightly backups dead; the surviving evidence counts 15. 0 log lines. Exit code 78.
Last updated:
01The quietest outage I ever had
The worst failure shape a system can have is the one that reports nothing, and I can date my conviction about that. All sixteen of my nightly database backup jobs were dead. Not degraded, not flaky. Dead, every night, with exit code 78, and not one of them said a word about it. No crash, no alert, no log line. The room was silent in exactly the way a healthy room is silent.
The one scheduled job that lived in a plain hidden folder ran perfectly the whole time. That single green light is why I never went looking. It is hard to overstate how convincing one working job is when you are deciding whether to audit sixteen of its siblings.
02Exit code 78, sixteen times a night
The mechanism fits in two lines. macOS refuses background jobs access to Desktop, Documents and Downloads unless you explicitly grant Full Disk Access, and a refused job dies before it can log a single word. My sixteen backup jobs pointed into a folder on the Desktop; the one survivor lived under a hidden dot-folder the wall does not guard. For the engineers: scheduled launchd agents hitting those protected paths fail with getcwd errors and exit 78 on chdir, so keep scheduled scripts and their data under plain paths, or grant access to a dedicated runner.
Two honesty notes before the lesson. First, my ledger records two separate silent outages of this same backup fleet. The permission wall killed the whole scheduled set at once; later, an unrelated one-line shell bug kept the nightly backups dead for weeks before a postmortem resurrected them. The weeks in this post's opening belong to that second entry. Two different killers, one identical symptom: silence. Second, the count is a note I wrote the night of, not a log. The rule file says sixteen, the plists I archived seventeen days later number fifteen, and a source comment from the week between says fourteen. No job wrote a line, so nothing on disk settles it. I am attributing sixteen, not asserting it.
03Why do broken jobs look exactly like working ones?
Picture it as arrows. Sixteen arrows leave the scheduler, hit an invisible wall in front of the project folders, and stop dead. One arrow takes a path with no wall on it and lands, every night, on time. From the outside you do not see arrows. You see one landed arrow and conclude the archery is fine.
The deeper trap is that the dead jobs and the living job produced identical output on my side of the glass: nothing. A backup that ran and had nothing worth saying makes exactly the sound of a backup the operating system killed on arrival. Failure arrived in the same tone as success, so I heard nothing and assumed everything.
04The second specimen: a queue that ate itself
The same shape turned up again in a completely different room, which is how I knew it was a shape and not an incident. My automation system's finished work queues up for my human verdict, and the count of recorded verdicts sat at 2 of a required 5 for days. Everyone, including me, read that as the human being behind on his homework.
The truth was better and worse. The sampler retired any item that carried any existing machine-written row, and 83 of the 85 rows in the pool were mechanical rederivations, analytics that the tool's own help text says are not human gold. The analytics pass had eaten the entire sampling pool, and the sampler reported 0 unadjudicated with no error anywhere. The queue was not waiting for me. It had been quietly made unreviewable, and it announced that in the same tone as a job well done.
05Zero out of what
Both stories collapse to one question, and it is the question I now put to every quiet system I own: zero out of what. A checker that finds no problems and a checker that is broken emit the same message. From the outside, zero problems found and the checker never ran are indistinguishable until something states the denominator.
So the fix is not more alerting, it is denominators. My backup probe prints how many repos it checked and how many of them failed, so a clean run reads as a count rather than an absence. Two denominators I described here as built are not built. The report still says nothing about how large each dump is, and the sampler computes its pool size and never prints it. I found both by opening the scripts while checking this page, which is the argument for opening the script. A zero with a denominator is information. A bare zero is a mood.
06Test the backup, not the intention
The schedule existing is an intention. The job firing is an intention. Only a restored file is a backup. Nothing in that chain counts until you pull on the last link, and pulling on it is cheap: open one dump, restore one table, read one byte count. I had weeks of intentions and zero backups, and the cost of discovering that early would have been about a minute a week.
The nightly schedule moved into CI, where the wall does not exist. The probe that watches the dumps on this machine still reads out of the folder the wall guards, which I only noticed while opening it to check this page, and what it does report every morning is a count. The rule that survives from the whole episode is short enough to keep: a green light must be a positive signal, never just the absence of a red one, and a safety net you have never pulled on is a rumor.
07How every number here was derived
Both source files are append-only, so re-running these will give larger denominators than the ones quoted.
grep -n "db-backup" ~/.claude/CLAUDE.md # LR#1: 16 *.db-backup agents into ~/Desktop/GitHub all failed 78; the ~/.sgnk job worked grep -n "exit 78" ~/.claude/CLAUDE.md # LR#1: exit 78 (chdir) under the TCC wall grep -n "for weeks" ~/.claude/CLAUDE.md # LR#55: nightly db backups silently dead for weeks (the second outage) grep -n "0 unadjudicated" ~/.claude/HANDOFF-aios-2026-08-01-MASTER.md # sampler reported 0 with no error grep -n "MIN_N=5" ~/.claude/HANDOFF-aios-2026-08-01-MASTER.md # human gold at n=2 of MIN_N=5 for days grep -n "83 of 85" ~/.claude/HANDOFF-aios-2026-08-01-MASTER.md # 83 of 85 rows were mechanical-rederivation, not human gold ls -1 ~/.sgnk/state/pruned-plists-20260714/*.db-backup.plist | wc -l # 15 archived plists; the rule file says 16, a 2026-07-10 source comment says 14 grep -n 'Desktop/GitHub' ~/.sgnk/bin/sgnk-backup-probe.sh # :24 the probe still reads the folder the wall guards cat ~/.sgnk/state/backup-alert.flag.seen # 2 repos failing at 2026-08-04T21:47:15Z, both dumps 45h old