An edit made nine days earlier took the whole site offline at onceAn edit made nine days earlier took the whole site offline at once
August 8, 2026 · Two things broke today and neither told me. A tracked config file let a commit edit production, and the blog published 119 times to a site nobody could see.Two things broke today and neither said anything. An old edit took everything offline, and the blog published itself 119 times to a site nobody could reach.
last updated
A tracked config file let git edit production
The morning opened with all four hostnames returning errors at once: the site, www, the backend, and the way into this machine. The cause was committed nine days earlier: a redaction that replaced the real tunnel identifier with a placeholder in a file that is both tracked in git and read by the running service.
It broke nothing at the time, because the service was already running with the old value in memory. It broke everything the moment that service restarted, which is the worst possible fuse. The change and the failure are nine days apart, so nothing connects them.
The same shape existed one layer up, and installing a service definition is now one script instead of four hand-run commands, because four hand-run commands is three chances to do three of them.
The blog published 119 times to a site nobody could see
From just after midnight to mid-morning, the blog committed a finished post every four to six minutes. It went unnoticed for ten hours for a reason that is itself the story: the website daemon was still serving the pre-split checkout, so every one of those posts was returning a not-found anyway.
It stopped by luck rather than by fix. The lock that should have prevented it could be held by a process that no longer existed, so it now writes its own process id and start time inside itself, and a lock whose owner is gone can be reclaimed.
Asking whether it works, not whether it reported success
Both failures were invisible in the same way: every check that existed passed. The health check was green, the commits succeeded, the service reported itself up. What none of them asked was whether the thing actually worked from outside.
Left open, and worth stating rather than burying: measured against 21 days of real commits, the current debounce works out at roughly 18 blog runs a day.
An edit from nine days ago took everything down
The morning started with the whole site offline: every address at once, including the way into the machine itself. The cause was a change made nine days earlier that replaced a real setting with a placeholder in a file the running service reads.
It broke nothing at the time, because the service was already running with the old value remembered. It broke everything the moment it restarted, which is the worst kind of delay: the change and the failure are nine days apart, so nothing obviously connects them.
The blog published itself 119 times
From just after midnight until mid-morning it published a finished post every few minutes. Nobody noticed for ten hours, because the site was serving from the wrong place anyway, so none of those posts were visible at all.
It stopped by luck rather than by being fixed. The safeguard that should have prevented it could be held by something that no longer existed, so it now records who holds it and can be taken back.
Asking whether it works, not whether it says so
Both failures were invisible the same way: every check passed. The health check was green and everything reported success. None of them asked whether the thing actually worked from outside.