A job killed two minutes before saving left all its work recoverableA job killed two minutes before saving left all its work recoverable
August 12, 2026 · Dispatched at 03:17, killed by the wall at 03:47, two minutes short of committing. The work was sitting in the working tree, half-landed twice over.A job was started at 03:17 and killed at 03:47, minutes before it would have saved its work. All of it was still sitting there, unsaved.
last updated
Lifted out of the tree of the session that died writing it
The ticket was dispatched at 03:17 and hit the thirty-minute wall at 03:47, about two minutes short of committing. Leaving that in the working tree was not a neutral state: the work had half-landed twice over, so the next job in that tree would have inherited it.
What it built is an append-only event log folded into a current holder per issue, so the live view and the changelog are the same function evaluated at different moments rather than two things that can disagree.
One thing about it was not the plan: the reconcile sweep turned out to be the primary route rather than the safety net, because the webhook subscription it depends on had lapsed. The page was right; it was just five minutes late, every time, and said so.
Restarting by asking, instead of killing whatever holds the port
Every job's prompt used to end with finding whatever holds port 3000 and killing it. That is unambiguous with one job running and dangerous with two. It aims at a port rather than at a job, so a restart meant for one deploy lands on a stranger's session mid-run.
Now a job writes a row into a requests table and waits. The worker reads it on its next tick and exits itself once nothing else is in flight, letting the supervisor relaunch it. The requester is exempt from its own wait, because a job asking to be restarted through has consented to it; a sibling has not.
Two copies of those instructions exist and nothing had ever enforced that they agree. Both could be rewritten and the whole suite would pass. There is now a test that reads both as text.
One job at a time became two
That is what the restart lock unblocks. The blocking precondition was a database lock fix from three days earlier, left deliberately to soak so it could be closed with a measurement rather than an assertion.
Rescued from a job that was killed halfway
A job started at 03:17 and was killed by a time limit at 03:47, about two minutes before it would have saved its work. All of it was still sitting there unsaved, and leaving it there was not harmless, because the next job would have picked it up as its own.
What it had built is a page showing who is holding which piece of work, built from a running log rather than a snapshot, so the live view and the history cannot disagree.
Restarting by asking, rather than by force
Jobs used to restart the server by finding whatever was using a particular connection and killing it. That is fine when only one job runs at a time and dangerous when two do, because it aims at the connection rather than at the job, so a restart meant for one job lands on someone else's.
Now a job asks, and waits. The server restarts itself once nothing else is running.