A message cut 101 characters short quietly threw away two ticketsA message cut 101 characters short quietly threw away two tickets

August 9, 2026 · Four things broke today and every one of them impersonated something else. The most expensive was a truncation that silently took two never-filed tickets with it.Four things broke today and each one looked like a different problem. The worst quietly threw away two tasks that were never written down.
last updated

A sign-off, cut 101 characters short

What gets stored of a session's output is the first 8,000 characters. This morning's session went past it, and the part that fell off the end was the part that listed two follow-up tickets it had decided needed filing. Neither was ever filed. Both were recovered by hand out of the raw log.

The shape is the one worth remembering: a cap applied to the head of a document, when everything anyone reads a session log for is at its end. The result is not an error, it is a slightly shorter success.

Ten of the eleven service definitions were never valid

The placeholder in the templates parsed as an unclosed tag, so ten of eleven files were not valid XML and never had been. What makes that survivable-looking is how a broken one announces itself: not at install, but silently, later, as a job that simply never runs.

The install script now checks the rendered file rather than the template. That is a different file from the one on disk, so not redundant with checking the source.

The queue learned which checkout each job belongs to

Every queued job now records the working tree it will run in, plus a per-tree ceiling. A ceiling over one partition buys nothing, and ten repos sharing one tree is one partition, so the partition had to come first.

It defused a real landmine on the way through: the claim step re-found the row it had just claimed by querying for the oldest dispatched one, which is correct only while at most one job can be running. With two, it returns the older still-running job and spawns a duplicate session against it.

A job landing on the fallback checkout now says why, because one repo had a valid grant, fell back anyway, and so read as a broken grant when the cause was something else entirely.

A note cut off, and two tasks lost with it

Only the first part of a session's output is kept. This morning's session ran past that limit, and the part that fell off the end was the bit listing two follow-up tasks it wanted filed. Neither was filed. Both had to be dug out of the raw log by hand.

The lesson: the limit was applied to the beginning of the document, when everything anyone actually needs is at the end. The result was not an error, just a slightly shorter success.

Ten setup files that were never valid

A placeholder in the templates made ten of eleven files technically broken, and always had been. The problem is how that announces itself: not when you install it, but silently later, as a scheduled job that simply never runs.

The queue learned where each job belongs

Each queued job now records which copy of the code it will run in, so two jobs in different projects can run at once without getting in each other's way. Doing that first caught a real trap that would have spawned duplicate work once more than one job could run.