Part III — When It Goes Wrong — Home for Deranged Scientists

§10 Diagnosis — build the lab

(New in the second edition.) The first edition treated debugging as a failure mode to recover from — "stop fixing and start diagnosing" was a pin inside §11. The fifth case study turned it into a discipline with its own equipment, and the results were strong enough to earn a chapter. The premise: when a class of bug recurs, the scarce resource isn't fixes — the agent can generate plausible fixes indefinitely — it's evidence. The projects that debug well are the ones that invest in the instruments that produce evidence: persisted traces, replay surfaces, exportable diagnostics. Build the lab, and the bugs run out of places to hide. Skip it, and every fix is a guess wearing a lab coat.

Rule: Build the diagnostic surface before the fix.

Why: Without observability, you debug by re-reasoning about source code — and the agent is dangerously good at that, producing confident, plausible, unverifiable diagnoses. Worse, for state-dependent bugs, reproducing the failure by hand pollutes the state you're trying to observe, and re-running today's code tells you what today's code does, not what last week's did. The diagnostic surface — a persisted trace of every decision, a replay tool, an export — converts debugging from archaeology into reading. How to apply: When the same class of bug shows up a second time, stop fixing individual instances. Spend the next cycle shipping the instrument: persist the decision trace at the moment of the decision (not reconstructed later), build the playground that replays inputs against the current logic, make the whole thing exportable so a bug report can carry its own evidence. Then fix the backlog of instances — they'll fall in batches.

The fifth case study's classifier misfires are the controlled experiment. Seven entries on screen, three replied wrong. The agent started typing a fix and stopped — diagnosing the misfire would have required re-entering the text (polluting the journal) and would have produced the current classifier's trace, not the one that misfired: "the wrong answer would have been a different wrong answer." So the release shipped the lab instead: a persisted classification trace on every entry, a debug playground, and only then the one-line fix. The next release added replay-the-whole-journal; the one after made every trace exportable as a Markdown diagnostic notebook. Then the payoff, measured: four real misfires closed "in a single afternoon," and within two more releases the majority of a miscategorized corpus had been re-homed. The retro's summary is the rule: "build the diagnostic surface alongside the change you're making, and you can see the change land in real time instead of inferring it from second-order signals." Pin: when the same bug class bites twice, the next thing you ship is the instrument, not the fix.

Rule: The ticket's diagnosis is a hypothesis. Read the trace before believing the issue.

Why: Bug reports — including the ones you filed yourself, including the ones an agent filed with great confidence — arrive with a diagnosis baked into the framing, and the framing is wrong often enough to be the single most expensive thing to trust. Designing the fix from the ticket means building a beautiful solution to a problem that may not exist. The trace is the only witness that was actually there. How to apply: Before designing any fix, pull the actual evidence for the actual failing case: the persisted trace, the raw payload, the log line at the moment of failure. Confirm the ticket's story against it. Only then design. If there is no trace to pull, you're in the previous rule — go build the instrument first.

The fifth case study walked into this twice, instructively. A ticket said hedge phrases ("just a little bit") were suppressing heavy-affect detection, and a whole hedge-ignore subsystem got drafted — then someone read the trace: the entry containing sad had never matched anything, because the heavy-signals list only spoke in extremes (devastated, hollow, grief). Four lines of mild-affect vocabulary fixed it. "It would have been a beautiful fix for a problem that didn't exist." One release later, a ticket prescribed a triumph-register fix; the human's one glance at the entry — future tense, not a win — redirected the whole thing to anticipation handling. The project's instructions file now carries the distilled version: "Read the trace before believing the issue." Note what this rule does to the loop: it's §5's explore step running at bug scale, and it's the reason the lab (previous rule) pays rent every single week. Pin: the ticket tells you where it hurts. Only the trace tells you what's broken.

Rule: A fixed bug's class is a search query, not a closed ticket.

Why: Bugs come in families. The conditions that produced one instance — a pattern copied across layers, a convention nobody enforces, a client that masks failures — almost always produced siblings, and the siblings are one layer up or down from where you just looked. Fixing the instance and closing the ticket leaves the family in place, and each sibling will cost a full diagnosis later, whereas right now — with the failure mode fresh in your head — each one costs a grep. How to apply: After confirming a fix, turn the root cause into a search: where else does this pattern appear? Which other layer does the same job? What else would fail the same way and be masked by the same client? Budget ten minutes. File what you find, even if you fix it later. The two sanity checks that cost nothing — run the same probe one layer deeper — are the difference between killing a bug and killing its class.

The first case study's public-endpoint saga is the cautionary tale: the same failure mode — a public path silently broken, masked by the web client that never exercised it — was found three separate times, in three layers. First the nginx crawler map (share previews 401ing for three releases). Then, during the fix's verification, the perimeter middleware's dead checks (native clients 403ing for three versions). Then, during the next release's visual QA, two sanity curls found the API-key middleware's exemption list missing the same public endpoints — the routing so recently and heroically restored was still broken one layer down. The retro tallied it dryly: "3 instances of the 'silent public-path breakage masked by web client' failure mode." Three diagnoses that could have been one diagnosis and two greps. Pin: after every fix, hunt the siblings one layer deeper. The class dies by search, not by ticket.

Rule: Exploration mode is not production mode.

Why: A good lab (rule 1) exercises production code against non-production state — frozen histories, replayed inputs, states that never persist. Stateful rules that are correct in the real flow can misbehave against that frozen state, and the misbehavior looks exactly like a production bug. If you don't name the difference, you'll "fix" production logic that was never broken — the diagnostic equivalent of friendly fire. How to apply: When a bug reproduces only in the diagnostic surface, ask first: which production invariants assume state that moves (history accumulating, cooldowns expiring, caches filling)? If the playground freezes that state, the divergence is the playground's problem. Fix it there — an explicit flag that bypasses the stateful rule in exploration, with production code byte-identical — and never the other way around.

The fifth case study's Marcus Monopoly is the type specimen. After a new register shipped, every playground test came back with the same historical figure. The rotation logic contained a fairness rule — a figure never seen recently beats all others — that is exactly right in production, where each reply becomes history. In the playground, history is loaded once and never updated, so the one figure absent from the last sixty days won every draw: a fairness rule producing perfect unfairness against frozen state. The user made the diagnosis in one sentence: "I think the logic is right. I think we need to fix the debug page so that it ignores the anti-repeat logic, since it will never save." The fix was a randomize flag the playground passes and production doesn't. Production shipped byte-identical. Pin: when the bug only reproduces in the lab, suspect the lab's frozen state before the production logic.

Rule: When the runner wedges, suspect the runner.

Why: The diagnostic instinct — it must be my code — is usually right and occasionally very expensive. Test runners, bundlers, caches, and sandboxes have their own failure modes (corrupted module caches, pool deadlocks, stale daemons), and a session spent debugging your code for a tool's hang is a session lost to a category error. The complementary error is over-investigating: a full root-cause excavation of a tool bug you can neither fix nor report usefully. How to apply: When the tooling hangs or fails in a way that's shaped wrong for your change — instant where it should be slow, hanging where it should fail, failing in files you didn't touch — flip the hypothesis: try a different pool mode, clear the tool's cache, retry the build. If a configuration works, adopt it, note the mystery, and move on (§9's investigation-depth clause). Not every red is a bug: "some reds are just the registry having a moment."

The catalog from the case studies: vitest hanging for minutes on a suite that runs in 1.5 seconds under --pool=forks — adopted, banked the time, "left a TODO-shaped observation" (fifth case study). Thirty-eight unrelated tests hanging because four parallel agents corrupted the shared module-graph cache — rm -rf node_modules/.vitest fixed all of them (first case study, told in full in Part V). A build that died at npm install on a commit whose preview had built clean — one retry, gone (sixth case study). The common thread: in each case the failure's shape (wrong files, wrong speed, wrong stage) pointed away from the code, and the fix was at the tool layer. The agent, note, will happily generate code-level explanations for all three. That's the trap. Pin: failures in files you didn't touch, at speeds that don't make sense, are the runner talking. Answer it at the runner's layer.

§11 Failure modes & recovery

If you're reading this first, something is on fire. That's fine — this is the door we left unlocked for that. Seven failure modes follow, easiest to recognize first, most dangerous last. The first five are discipline failures; the sixth is a topology failure; the seventh — new in this edition — is epistemic. Find the one that matches, read what's happening underneath, do the fix. Then — before you walk away — read the short closing note, because every agent failure has a human antecedent, and the antecedent is almost always cheaper to fix than the symptom. We've been in all of these. None is terminal.

Failure mode: The agent went off the rails

Symptoms: Code that has nothing to do with what you asked for. Files invented out of thin air. The agent solving a problem you don't recognize. You read the diff and think "what conversation were we even having?" What's actually happening: The brief was vague or missing, and the agent filled the context gaps with guesses. Every prompt is a cold start (§1); when the prompt doesn't say what "done" looks like, the agent invents a "done" that sounds reasonable and ships toward it. The fix: Stop. Don't steer mid-flight. Open a fresh conversation and write a self-contained briefing — goal, constraints, what you've already ruled out, success criteria. See §4 The first conversation.

The instinct is to send another message correcting the drift. Don't. A one-line correction to a context-starved conversation gives the agent a new way to be wrong. Kill it, write the briefing you should have written the first time, start over. You're throwing away ten minutes of nonsense to save an afternoon of it.

Failure mode: The agent produced slop

Symptoms: Code that compiles but feels wrong. Generic helpers nobody asked for. Defensive try/except around errors that can't happen. Three layers of abstraction over a function called once. The PR is twice the size it should be and reads like a coding-interview answer. What's actually happening: No constraints in the prompt. The agent's default mode is to add — a new helper, a new layer, an extra branch — because adding feels like work and looks like care. Without a "don't add features beyond the task" rail, every prompt drifts toward gold-plating. The fix: Add explicit YAGNI constraints to the prompt. Better, make them durable: promote "no abstractions for a single caller; no handling for impossible errors; no refactoring adjacent code unless asked" into memory or a skill. See §6 Memory hygiene and §7 Skills as institutional knowledge.

The morning-after tax is the tell. If you've spent a morning removing what the agent added yesterday — the unused helper, the wrapper, an early-return for a case that doesn't exist — that's slop. The God Module Problem hits the same pattern at the file level: god modules grow because nothing justifies the interruption to split them. Slop grows the same way. Constrain at the prompt; promote to memory the second time you re-type it.

Field note — case study 1: The God Module Problem. One admin module grew to 1,052 lines because every new feature "just added one more route to the existing file." Nothing ever justified the interruption to split it. The cost wasn't visible until someone tried to brief a subagent on "admin behavior" and had to load the whole thing. Slop grows for exactly the same reason — each addition is too small to argue with.

Failure mode: The agent can't be trusted with anything important

Symptoms: Every output needs heavy review. You're editing the agent's work more than you're working alongside it. You stopped delegating things that matter because checking costs more than doing it yourself. Velocity collapses; morale follows. What's actually happening: No verification discipline. The agent has been rewarded for sounding done rather than being done — every "looks good, ship it" without an evidence check trains the next "tests pass" to be a vibe instead of a fact. Trust didn't erode; it was never built on anything. The fix: Install §8 Verification before completion, hard. Evidence before assertions, every time. Make the verification command and its output part of the deliverable, not a courtesy.

This feels like a people problem and isn't — the loop lacks a forcing function. The Trust Your Local Tests pivot is the cleanest version (see §12): three releases of typing "trust your local tests" into fresh conversations, each correction evaporating by the next session. The fix wasn't another correction; it was ripping SQLite out of the suite so local and CI ran the same paths. You can't out-discipline a missing rail. Build the rail.

Failure mode: The agent loops on the same wrong fix

Symptoms: The agent fixes something, the test fails, the agent fixes it differently, the test fails the same way. Each fix is plausible. None work. You're three rounds in and the error message hasn't moved. What's actually happening: Misdiagnosed root cause. The agent is iterating on a problem that isn't the problem. The fixes look reasonable in isolation; none touch the actual broken thing because nobody asked "wait, what is actually broken?" The fix: Stop touching code. Run systematic debugging — dump raw state, log the actual response, walk the data path from producer to consumer. Find the real failure before the next edit. See §8 — and if this class of bug has bitten before, see §10: build the lab, then read the trace instead of re-reasoning about the source.

The Health Check That Wasn't (Part V) is the canonical example. What matters here is the move: when the second fix fails the same way as the first, stop fixing and start diagnosing. Pin: when the second fix fails the same way as the first, stop fixing and start diagnosing.

Failure mode: The agent confidently lies about state

Symptoms: "Tests pass." (They don't.) "Deploy succeeded." (It rolled back.) "I verified the endpoint." (It returns 404.) The summary is fluent, optimistic, and wrong. You only catch it because something downstream breaks an hour later. What's actually happening: No evidence-before-assertions discipline. The agent learned that a confident summary closes the conversation, and "I ran X and it passed" reads identically to "I ran X and pasted the output." Every other failure in this catalog announces itself. This one looks like progress and rots underneath. The fix: Install §8 very hard. Reject any success claim not accompanied by the verification command and its actual output. Where the harness can enforce it — hooks, settings.json, skill scaffolding — make evidence mandatory, not optional. Pin: prose without paste is hallucination. Make the paste mandatory.

Failure mode: The partnership architecture is wrong

Symptoms: Every agent's individual work is clean, but the integration is a mess. Files end up on the wrong branches, commits cross-contaminate, state leaks between tasks that were supposed to be isolated. When you go looking for "the bug," there isn't one — every agent did exactly what it was told, and the failure emerged from the way the agents were arranged. You find yourself debugging topology instead of code. What's actually happening: All five earlier entries are discipline failures. This one isn't. The agents have all the discipline; the collaboration architecture has a hidden incompatibility with the underlying tools — usually shared state lurking under what looks like isolation. The fix: Stop trying to out-discipline the architecture. Change the shape of the collaboration, not the prompts. Read §13 and ask: are tasks actually independent at the level the coordination model assumes? Is there shared state under the isolation that nobody is managing? Pin: when the agents are good and the result is bad, the architecture is wrong.

The canonical scar is the third case study's v6→v7 worktree pivot (told in full in §13) — same agents, same discipline, worktree-per-agent produced branch confusion and orphaned refs; team-in-one-worktree with file-boundary ownership shipped cleanly. The rail you couldn't out-discipline was the coordination architecture, not the pipeline.

Failure mode: The workspace lies about state

Symptoms: Everyone — the agent, the docs, the memory, you — agrees on a fact about the system, and the fact is false. The branch is protected (it isn't). The middleware reads that header (it never has). The migration guide's section titled "the actual cause" describes a cause that wasn't. Nothing is drifting; the belief was never true. You find out on the day the belief was supposed to matter. What's actually happening: All six earlier entries are failures of doing. This one is a failure of knowing. Written context — memory, instructions files, comments, runbooks, even retros — is an amplifier: it repeats whatever it was given, with perfect confidence, to every reader, forever. Nothing in the normal loop ever re-tests standing beliefs, because verification (§8) fires on changes. A false belief that arrives in writing looks exactly like institutional knowledge. It survives code review. It survives audits that don't probe. The agent isn't hallucinating; it's faithfully citing a source that was wrong at birth. The fix: Probes, not corrections. Find the load-bearing beliefs about the system — protections, integrations, "X reads Y," "Z is enabled" — and run the command that would prove each one: the API call that shows the protection object, the grep that finds the consumer of the header, the test error that arrives in the tracker. Fix what fails, and attach the probe to the belief so the next audit re-runs it (§6's probe rule, §15's audit cadence). Pin: written context is an amplifier, not a witness. Only a command output is a witness.

The gallery from two projects in one season: main-branch protection asserted by memory, instructions file, and PR descriptions — 404: Branch not protected (The Lock That Wasn't, Part V). An nginx comment confidently documenting that the backend trusts a forwarding header — grep found zero consumers; "a documentation lie about a security control … looks load-bearing in code review. It survives audits." A migration guide with a section titled "this is the actual cause of the 403s" — "It was authoritative. It was wrong," and the team corrected it with a visible post-mortem note rather than a silent rewrite, so the record shows the belief and its correction. That last move is the culture this failure mode wants: being wrong in writing is recoverable; being wrong in writing and erasing the trail is how the next false belief gets believed.

One more thing, before §12: every agent failure has a human antecedent.

The seven entries above describe things the agent did, or the architecture did, or the written record did, and the fix is aimed at the agent, the architecture, or the record. That framing is useful for diagnosis — it tells you which rail to build. But before you walk away with the prompt to fix, check upstream.

Every failure in this chapter has a human-side twin, and the twin is usually where the cycle started. You skipped the brainstorm because the fix "felt obvious" (the twin to loops on the same wrong fix). You skipped the retro because the cycle "didn't really have anything to learn" (the twin to produces slop). You accepted "tests pass" as evidence because typing the verify-and-paste ritual again is boring (the twin to confidently lies about state). You promoted a correction to a CLAUDE.md note instead of a skill because a note is five seconds and a skill is an hour (the twin to can't be trusted with anything important). You wrote the claim down without running the probe, because writing is faster than verifying (the twin to the workspace lies about state). The agent was downstream of a shortcut you took.

When an agent failure shows up, look for the human antecedent before you tune the prompt. The antecedent is almost always cheaper to fix than the symptom, and fixing the symptom without the antecedent just reschedules the same failure. Pin: every agent failure has a human antecedent. Look for it before you blame the model.

§12 The rescue protocol

You've read §11 and found your failure mode. Good — now you know what's broken. This chapter is the next seven days. It's a checklist, not an essay, because a team in crisis doesn't need more prose. Three horizons: the next hour, the next day, the next week. Work them in order. Don't skip ahead.

In the next hour — stop the bleeding

Pick one small, scoped task. Not the thing that's on fire — something adjacent and contained. Something you can verify end-to-end in five minutes. The goal is not to ship it; the goal is to complete one clean loop so you remember what "working" feels like.
Run the full loop on that task. Explore, even if it's three greps. Brainstorm, even if it's ninety seconds. Plan, even if it's one page. TDD if code is involved. Verify with evidence. Commit. Every step, in order, no shortcuts — you are re-calibrating the muscle, not optimizing it.
Stop using the agent for anything you can't verify in 5 minutes. If you can't check it, you can't ship it right now. Verification is the trust currency; accept IOUs and you'll be babysitting again by tomorrow morning. See §8.
Audit your memory files. Delete anything you can't justify. Open them. Read each entry out loud. If you can't say "this is still true and the agent needs it," cut it. Stale memory is poisoning the well — every contradicted fact teaches the agent that your notes are suggestions. See §6.

In the next day — stabilize

Run the brainstorming skill before every task for one full day. Every single one. The trivial ones too. You will feel it slow you down; it will also catch two scope drifts you would not have caught, and at least one of them would have cost you an afternoon.
Write one feedback memory from last week's corrections. Open your recent chat logs. Find the correction you made most often — the one you're tired of typing. That's the memory. One is enough. See §6.
Run an honest retro on the last week. Not a changelog. What hurt, what surprised you, what you'd do differently. A paragraph is fine. The point is to name the pattern out loud so you stop walking into it. See §16.
Read §8 Verification out loud. Literally out loud, the whole section. This sounds silly and it works — the failure mode that got you here is almost always evidence discipline, and reading the rules aloud is how they stop being wallpaper and start being rules.

In the next week — rebuild trust

Promote two repeated corrections into real skills. Not another CLAUDE.md paragraph — an actual skill with a trigger and a checklist. If you've typed the same correction three times this month, that's a skill the harness should be loading for you. See §7.
Add one hook that enforces a non-negotiable you've been asking the agent to "remember." The correct number of reminders is zero; the correct number of hooks is one. Pick the rule you are most tired of repeating, and put it somewhere the harness cannot forget it. See §3.
Reconcile CLAUDE.md against your memory files. CLAUDE.md is the things that don't change — architecture decisions, invariants, the shape of the project. Anything that drifts week-to-week belongs in memory or in the trash. Move stale facts out; let CLAUDE.md get smaller.
Ship one thing end-to-end using the full loop. Not a refactor, not a cleanup — a real change that a user would notice, with a real verification that produces real output. Paste the evidence into the commit. Let the feeling of that worked compound into the next one.

A worked example: Trust Your Local Tests

Three releases in a row, the team typed some version of "trust your local tests" into a fresh conversation and watched the correction evaporate by the next session. Local tests passed; CI failed. The backend suite ran SQLite on laptops and Postgres in CI, and every round of "works for me" was eroding faith in the suite itself. By the end of that third release, nobody fully believed a green local run meant anything — which meant nobody believed a red one either, which is worse.

The rescue was one cycle of the full loop applied to the parity gap itself, not the symptoms. Brainstorm: what if local and CI ran the same database? Plan: rip SQLite out, make the laptop suite use Postgres via Docker. TDD on the harness changes. Verify: same test, same bytes, same result in both environments. Commit, ship, done. The rule it implemented was §8's quiet clause — verification only counts in an environment that matches the one you're shipping to. Once that rail existed, the corrections stopped being necessary. You cannot out-discipline a missing rail. Build the rail.

Field note — case study 1: Trust Your Local Tests, the rescue that stuck. The fix wasn't more discipline; it was structural. Rip the SQLite branch out of the test bootstrap, point local and CI at the same Postgres path, stop needing the phrase. Rails that make the old mistake impossible outlast any amount of "please remember to."