The Brilliant Employee
The Brilliant Employee · 06

Every Nightly Backup Was Dead. I Still Cannot Tell You How Many There Were

Scheduled jobs died at startup under a macOS permission wall, before writing a line. With no logs, three of my own records disagree on the count and nothing can settle it.

AuthorSagnik Mitra
Published2026-08-04
Reading timeabout 4 min
Deck11 slides
Download PDF

What is inside

  1. Every backup died, and the outage deleted its own evidence
  2. The mechanism, which is not the news
  3. A dead job and a healthy one produce identical output
  4. The fix: give every quiet report a denominator
  5. How to check your own scheduled jobs
  6. What this buys, stated plainly
  7. How every number here was derived

All my nightly database backup jobs were dead, every night, with exit code 78. None of them logged a word, because they died before they could. macOS refuses background jobs access to Desktop, Documents and Downloads without Full Disk Access, and my jobs pointed at a folder on the Desktop. One job survived because it lived under a hidden folder, and that single green light is why I never looked. With no logs, my own records now say sixteen, fifteen and fourteen, and nothing can settle it.

Key facts. My note says 16 dead jobs; the archived config files count 15; a source comment says 14. Log lines written: 0. Exit code 78.

Last updated:

01Every backup died, and the outage deleted its own evidence

My nightly database backups run as scheduled background jobs. All of them were dead. Not degraded, not flaky. Dead every night, with exit code 78, and not one said a word about it.

I cannot tell you how many there were. My rule file, typed the night it surfaced, says sixteen. Seventeen days later, the config files I archived number fifteen, and from the week in between, a source comment says fourteen.

Since no job ever wrote a line, nothing on disk arbitrates between the three. Nothing ever will. The incident took out the only witness to itself.

One scheduled job ran perfectly the whole time, the one living in a plain hidden folder. That single green light is why I never went looking. One working job is enormously convincing when you are deciding whether to audit the rest.

You can rent the model. You cannot rent the part that tells you it was wrong.
how many jobs died, according to my own records
rule file, written that night
16
archived config files, 17 days later
15
a source comment from the week between
14
log lines available to settle it
0
Three counts, one incident, no arbiter. This is what a silent failure costs after the fact.

02The mechanism, which is not the news

This part fits in two lines and it is well-trodden ground. macOS refuses background jobs access to Desktop, Documents and Downloads unless you explicitly grant Full Disk Access. A refused job dies before it can log a single word.

My backup jobs pointed into a folder on the Desktop. The survivor lived under a hidden dot-folder, which the wall does not guard.

For the engineers: scheduled launchd agents, meaning jobs run by the macOS background scheduler, hitting those protected paths fail with getcwd errors and exit 78 on chdir. Keep scheduled scripts and their data under plain paths, or grant access to a dedicated runner.

Apple documents the wall. The failure signature is widely known. If you came for the mechanism you can stop here. What follows is the specimen, laid out with the parts that do not reconcile left visible.

the scheduler starts the job on timeevery night, exactly as configured
the job tries to enter a protected folderDesktop, Documents or Downloads
the OS refuses and the job exits 78before any logging code runs
nothing is written anywhereno log, no alert, no dump
The job dies at step three. Every logging line it contains sits at step four.

03A dead job and a healthy one produce identical output

Picture arrows. Sixteen leave the scheduler, hit an invisible wall in front of the project folders, and stop dead. One takes a path with no wall and lands, every night, on time. From outside you do not see arrows. You see one landed arrow and conclude the archery is fine.

On my side of the glass, the dead jobs and the living job produced identical output. Nothing.

A backup that ran and had nothing worth saying makes exactly the sound of a backup the operating system killed on arrival. Failure arrived in the same tone as success, so I heard nothing and assumed everything.

The same shape turned up in a different part of the system a few weeks later, in a review queue that reported zero remaining after an analytics job had eaten it. Different cause, identical symptom, which is how I knew this was a shape and not an incident.

a healthy quiet job
ran
yes
found problems
none
wrote to the log
nothing
you observe
silence
a job killed at startup
ran
no
found problems
never looked
wrote to the log
nothing
you observe
silence
Only the last row is visible to you, and it is the same row in both columns.

04The fix: give every quiet report a denominator

What falls out of this is denominators. Make every quiet report say zero out of what, so that a clean run reads as a count rather than as an absence.

A checker that finds no problems and a checker that is broken emit the same message. The gap between them closes the moment something states the total. A zero with a denominator is information. A bare zero is a mood.

That advice is old and it is not mine. What I can offer is the audit of whether I actually took it, and the answer is partly. Today my backup probe does print how many repositories it checked and how many failed. But two other denominators I had described as built are not built: nothing in the report says how large each dump is, and the sampler computes its pool size and then never prints it.

Both fell over when I opened the scripts while checking this page, which took less effort than writing the sentence claiming they worked.

what it printed before
output
(nothing)
means
healthy, or dead
actionable
no
what it prints now
output
0 failed of 14 checked
means
healthy, unambiguously
actionable
yes, if 14 drops
The denominator is the entire fix. It is also the part I had claimed and not built, twice.

05How to check your own scheduled jobs

Having a schedule is an intention. Firing the job is an intention. Only a restored file is a backup.

Nothing in that chain counts until you pull on the last link, and pulling on it is cheap: open one dump, restore one table, read one byte count. I still had weeks of intentions and zero backups, where finding out early would have cost about a minute a week.

1List every scheduled job and the path it writes to. Flag anything under Desktop, Documents or Downloads.
2Check each job's last exit code, not its last log line. A job that never logged has no log line to check.
3Make each job print a count, not a silence: how many things it checked, how many failed.
4Restore one file from one backup. Not the schedule, not the dump size. The restore.
Step 2 is the one that would have caught this on night one.

06What this buys, stated plainly

One rule survives the whole episode and it is short enough to keep. A green light must be a positive signal, never just the absence of a red one.

I moved nightly scheduling into a hosted runner, where the wall does not exist. My probe that watches the dumps on this machine still reads out of the folder the wall guards, which I only noticed while opening it to check this page. What it does report every morning is a count.

The honest summary is that the mechanism here is old ground. The specimen is not. Almost nobody publishes what a silent outage looks like afterwards, when three of your own records disagree and there is no evidence left to settle it.

1A green light must be a positive signal, not the absence of a red one.
2Every quiet report needs a denominator: zero out of what, counted by whom.
3A job that dies before logging leaves no evidence that it died.
4A safety net you have never pulled on is a rumour.
Silence is a bad thing to take on trust.
A backup you have never restored is a schedule, not a backup.

07How every number here was derived

These commands read live state, so re-running them will give different numbers from the ones quoted here, usually larger. That is the point of printing the commands rather than only the conclusions.

# LR#1: 16 *.db-backup agents pointing into ~/Desktop/GitHub all failed 78; the ~/.sgnk job worked
grep -n "db-backup" ~/.claude/CLAUDE.md

# LR#1: exit 78 (chdir) under the Full Disk Access wall
grep -n "exit 78" ~/.claude/CLAUDE.md

# LR#55: a separate one-line shell bug kept the nightly backups dead for weeks
grep -n "for weeks" ~/.claude/CLAUDE.md

# the three counts that disagree, in the order they were written
grep -rn "16 \*.db-backup\|15 plists\|14 agents" ~/.claude/CLAUDE.md ~/.sgnk/bin/

# your own scheduled jobs, and which of them point at a protected path
launchctl list | grep -v com.apple
ls ~/Library/LaunchAgents/*.plist | xargs grep -l "Desktop\|Documents\|Downloads"

# the denominator that exists today
~/.sgnk/bin/backup-probe.sh --report
All episodes