A killed job left 15 bytes of explanation, so jobs now explain themselvesA killed job left 15 bytes of explanation, so jobs now explain themselves

August 13, 2026 · Yesterday's session was killed two minutes short of committing four thousand finished lines, and signed off with the words Execution error.Yesterday a job was killed minutes before saving hours of finished work, and all it left behind was the words Execution error.
last updated

A failed run leaves evidence now, not a sentence

The starting point was worse than it looked. Yesterday's session was killed about two minutes short of committing four thousand lines of finished work, and the entire machine-readable record of those thirty minutes was fifteen bytes: Execution error. Across the 452 logs before it the median is 2,352 bytes, and every one is a final answer rather than a trace, because the tool prints only its last message, and a killed session never writes one.

Sessions now stream their whole transcript, measured at 34,380 bytes on a real killed run against fifteen. The session id is stamped on the task's own row, write-once, so a failure can be traced to its transcript instead of matched to one by timestamp and guesswork, which at two concurrent jobs is a coin flip.

On top of that, four read-only lookups run on the completion path and their finding becomes a block in the comment: whether a restart touched the run window, whether the tree is dirty, what the log's tail says, and whether the process is still alive. What that replaces is the only root-cause analysis this codebase had: the literal words “almost certainly killed by a server restart”, asserted about every orphan and verified for none.

Classing it, and retrying exactly once

The diagnosis explains one run in prose; a classifier reduces it to one of six classes and asks the question four lookups cannot: whether the failure revealed something separate worth its own ticket.

The only piece that acts is a retry, and its scope is deliberately a property of the outcome rather than a list of classes: only a run that never emitted a result line at all is a candidate. A session that spoke reached its own end, and re-running it overrules a report rather than recovering from an accident.

The board called me unassigned a minute after its webhook went live

Switching the issue webhook on made yesterday's board actively wrong inside a minute: it recorded a ticket moving away from me while the tracker correctly said otherwise. The ingest could not resolve identities from a cold cache, so it placed a real person as nobody.

A failed job leaves evidence now

Yesterday a job was killed about two minutes before it would have saved hours of finished work, and the entire record of those thirty minutes was fifteen characters: Execution error. The tool only prints its final message, and a job that gets killed never writes one.

Jobs now record everything they do as they go: about 34,000 characters on a real killed run, against fifteen. Each job also stamps which transcript belongs to it, so a failure can be traced rather than matched up by guesswork.

On top of that, four quick checks now run whenever something fails, and what they find goes into the note on the task: was there a restart, was there unsaved work, what does the log say, is it still running. What that replaces is a single sentence that was asserted every time and checked never.

Trying again, exactly once

A failure is now sorted into one of six kinds, and only one kind is ever retried: a job that never reported anything at all. A job that did report something reached its own conclusion, and re-running it would be overruling its answer rather than recovering from an accident.