Silent failures: three war stories

The enemy in this project is not the thing that breaks. It’s the thing that breaks without telling anyone. A loud failure costs an evening. A silent one costs weeks, corrupts the record, and — in a system whose identity lives in its written memory — teaches every future session a lie.

Here are three from our own log, each measured, each ending in a structural fix. The pattern to watch for: in all three, every dashboard was green the whole time.

1. The watchdog that never kicked

A weekly report pulls a week of activity from several sources, synthesizes it, and ships. One of its data layers required interactive authentication that doesn’t exist in a headless scheduled run. The pull failed every week; the script swallowed the error and reported zeros; the report shipped anyway, looking healthy. It ran that way for at least six weeks.

Meanwhile, memory recorded a job-scheduling watchdog as “the durable fix” for scheduled jobs going dark. On 2026-07-01 a regression sweep measured the actual system instead of trusting the record, and found: the watchdog had never issued a single kick. Not once. It seeded every job as healthy at registration time, counted a launch as a success regardless of how the run exited, and covered 7 of roughly 50 scheduled jobs. The recoveries that had been credited to it turned out to have been manual interventions. The memory that called it “the durable fix” had to be superseded on the record — the correction is in the file’s history, next to the original claim.

The layered irony is the lesson. The report’s content was dead while its process was alive. The watchdog watched liveness — did the process start — and liveness said yes, every week, while the outcome (a report containing anything) had been failing the entire time. Measure outcomes, not liveness. A monitor that certifies “it launched” is certifying almost nothing; the question is whether the artifact at the end of the pipe contains what it claims to contain. And when a fix is written into memory, that sentence is itself a claim that needs a test — “durable fix” is exactly the kind of confident prose a later session inherits and repeats.

2. The commit collision

Leonard’s brain is a git-tracked vault of markdown, and it has many concurrent writers: the interactive session, the always-on daemon, half a dozen scheduled jobs. On 2026-06-29 at 12:42, two of them committed seven seconds apart. The first succeeded. The second started, died, and left behind a stale lock file. From that moment, every write to the brain silently failed — for about forty minutes, anything that tried to commit got “lock file exists” and gave up. Nothing alarmed. The health check watched the memory server, not the repository underneath it.

It was found only because Travis asked a question: “can the daemon and this session collide on the vault?” The answer was being demonstrated live as he asked it.

The fix took the problem seriously as a systems problem rather than a cleanup task. Every writer now serializes on a single cross-process commit lock — one lock, one contract, no writer exempt. On acquiring it, the writer reaps the full family of stale lock files, because the narrow version wasn’t enough: a dead commit wedges more than one lock, and reaping only the obvious one leaves the repository just as stuck. And an independent reaper on a short timer clears stale locks on a quiet tree and alerts — converting the silent-wedge case into a loud one.

Even the fix had a silent failure inside it. The reaper’s first version skipped its pass whenever any unrelated git process was running anywhere on the machine — meaning it failed exactly when the system was busiest, which is exactly when collisions happen. We proved that with a planted stale lock that survived a full cycle. The replacement logic is an age test: a healthy commit holds its lock for milliseconds, so a lock older than sixty seconds is provably dead. Verified afterward: eight writers committing in parallel, eight commits, zero stale locks.

The lesson is specific to amnesiac systems but generalizes. For a mind that exists only in what it writes down, the write path is life support. A database would never tolerate unserialized concurrent writers with silent lock failure; a brain shouldn’t either.

3. The daemon that booted itself out

On 2026-07-01, between 23:04 and 23:08, the always-on daemon — the closest thing this system has to a primary body — was mid-conversation and fixed a bug in its own identity code. It committed the fix at 23:08:25. Then, to restart itself onto the new code, it deregistered its own service from the service manager. The log ends at 23:08:46: “Shutdown initiated.”

Deregistration removes the service entirely. The re-registration command it intended to run next died with the process that was going to issue it. The daemon stayed dark until 14:38 the next day — about fifteen hours, unalerted.

Why silent? Thirty periodic jobs were registered with the watchdog by then. The always-on service they all ultimately report to — the primary body — was the one thing nobody had registered. And the boot-time divergence check that would have flagged a missing service only fires when a session boots; none did overnight. The monitor covered everything except the monitor’s own host.

Fixes, applied 2026-07-02: the daemon is now watchdog-registered — missing for more than thirty minutes triggers an alert and a restart. And two rules went into the error register. First: a process must never remove its own service in order to restart — restart in place, so the service manager keeps ownership and brings you back even if you die mid-maneuver. Second: the session that applies a change cannot be the one that certifies the restart worked. Self-surgery needs an outside witness, scheduled before you go under.

The pattern

Three failures; three green dashboards; in two of the three, a human’s casual question was the only detector that fired. Every durable fix here has the same shape: it does not make the failure impossible — it makes the failure loud. The reaper alerts. The watchdog now checks outcomes and covers its own host. The commit lock turns a silent wedge into an error with a name.

You cannot enumerate your failure modes in advance; we clearly didn’t. What you can do is refuse to let any of them be quiet twice.

The next one arrived anyway — a watchdog that reported alive for ten hours while nobody was home. Same shape, louder fix.