Part V — Field Notes

Every rule in this playbook has a bruise under it. The entries below are where the bruises came from — compact re-tellings of the bugs that taught us the rules, each one naming the chapter it grounds. Read them in any order; each is self-contained. The headlines are whimsical on purpose, because a bug with a nickname is a bug you remember.

The stories are grouped by the kind of problem they illustrate, so a reader scanning for "I'm having a verification problem" or "my parallel agents are stepping on each other" can jump to the right neighborhood first. Within a group, they're in no particular order.

Verification & feedback loops

Stories where the test was green, the status was "ok," and the thing was broken. The bruises here all trace back to verifying the wrong surface or trusting a signal that was answering a different question than the one you were asking.

The Health Check That Wasn't (v5.11.0)

What happened: Three deploys in a row, git_sha came back unknown in the health response after every release. We rebuilt the Docker image pipeline four different ways — shell export of GIT_REV, compose env file, compose override, --build-arg — and each fix deployed cleanly and "didn't work." What we thought was happening: The build-arg plumbing was dropping GIT_REV somewhere between the YAML and the container, and we just had to find where. What was actually happening: The verification curl was hitting /health, which returns {"status":"ok"} and has never had a git_sha field. The full health JSON lives at /api/v1/health. The producer was never broken. We spent three deploys fixing a hole that wasn't there because the verifier was looking through the wrong window. The lesson: §8 — health checks must check health, and verification curls must hit the endpoint that reports it. (See DevOps Playbook Gotcha #8.)

Verify What You Shipped, Not What You Built (v5.9.0)

What happened: The deploy script reported success. The new container had started. Every status check was green and every log line said "ok." The live site was still serving the old version. What we thought was happening: A browser cache, a CDN delay, something between the user and the server — not the deploy itself. What was actually happening: An orphan container from the previous compose definition was still holding the port. The new container was running and healthy and completely unreachable from the internet, because nginx had never flipped to it. "I started the container" had become a stand-in for "the container is serving traffic." The fix was one line of verification: one curl against the public health URL, comparing the response body to the SHA we just built. The lesson: §8 — evidence before assertions, against the URL the user will actually hit.

Trust Your Local Tests (v5.7.0)

What happened: For three releases running, the backend test suite passed on every laptop and failed intermittently in CI. Nobody trusted "local green" anymore — every PR got a manual "but did it pass in CI?" even after it had passed in CI. What we thought was happening: Flaky tests. A Redis timing issue. Something environmental we'd catch eventually. What was actually happening: Local used SQLite via a is_testing branch in database.py; CI used Postgres. Every release, the gap introduced a new dialect-specific failure, and every release we patched the symptom. The rescue was a single cycle of the full loop applied to the gap itself — rip the SQLite branch out, make conftest.py truncate a shared Postgres engine between tests, force every environment through the same path. The lesson: §8 — verify in the environment that matters, and §12 — you cannot out-discipline a missing rail.

The Cache That Worked Locally And Lied In CI (v5.4.0)

What happened: We refactored the cache layer so Pydantic models flowed through Redis, shipped the PR with a green local suite, and watched an image-generation pipeline explode in CI the moment the deploy ran. What we thought was happening: A Redis connection issue, or maybe a dependency version mismatch in the CI image. What was actually happening: Someone had written json.dumps(model, default=str) as the serializer. default=str does not dictify a Pydantic model — it calls str() on it and produces the model's repr, a string that looks like Python source. Local was green because local used an in-memory cache with no serialization path at all; the model went in as an object and came out as an object. CI had actual Redis, which needed JSON, which exposed the bug. The lesson: §8 — if your local doesn't run what CI runs, your local green is a lie.

The Doorbell That Never Rang (v4.2.0)

What happened: Audit logging shipped. Backend tests green, frontend tests green, a beautiful admin page that rendered an audit table. The table had zero rows in it. Every admin action completed successfully and left no trace. What we thought was happening: A query filter bug, or a timezone off-by-one making events land outside the default window. What was actually happening: Nothing in between the two halves emitted an audit event. Backend exposed the read endpoint. Frontend rendered the list. No service call anywhere in the admin routes actually fired the emit. Both halves were tested in isolation; nobody had wired the doorbell to the button. A thirty-second click in a real browser would have caught it. The lesson: §8 — "tests pass" is not "feature works." Open the thing and use it.

The Comparison That Quantum-Superposed Into Nonexistence (case study 1)

What happened: A "Comparison of the Day" feature shipped to the homepage. The component was implemented correctly, the data fetched correctly, the unit tests passed, the integration tests passed, the deploy was clean. It never appeared on the page. What we thought was happening: A caching or CDN bug swallowing the response on first paint. What was actually happening: The component was wrapped in {!isLoading && <Feature />}, and on the homepage's particular data path isLoading never resolved to false because of a stale-cache race condition in a sibling component. The feature rendered into a virtual DOM that nobody ever painted. If The Doorbell is about missing wiring (the event never fired), this is the inverse: the bell rang, perfectly, into a room nobody could see into. The lesson: §8 — a green test suite tells you the parts work; only a real session in the real browser, on the real page, with the real data, tells you the user can use the thing.

Two Hotfixes in One Release, Same Gap (case study 3)

What happened: A data-layer migration from manual useEffect/useState to TanStack Query shipped with 161 unit tests green and broke cache-hit paint within 24 hours of release. Its stablemate, shipped the same day for the same reason, was a touch-vs-mouse guard that had been written and tested on desktop only and silently blocked single taps on iPad Safari. What we thought was happening: Both changes were mechanical — data layer swap and an input-event refactor. Unit tests covered the data behavior and the handler logic. Nothing to see here. What was actually happening: The TQ bug: a side effect moved inside queryFn never ran when TQ served cached data. The iPad bug: the guard had never been exercised on an actual touch device. Both failures slipped a full green suite because no test exercised the interaction pipeline on the real substrate — cache-warmed data + real click, or real gesture on a real device. A single Playwright click against a pre-warmed cache would have caught the TQ bug in five seconds; a real tap on a real iPad would have caught the other. The lesson: §8 — when you're swapping a foundation layer (data layer, auth, router, state), the test gap is always on the interaction side, because the unit tests were written for the old substrate's assumptions.

The bufconn Gap (case study 2)

What happened: One protocol adapter had full unit-test coverage using an in-memory connection that bypassed network transport. Every test passed beautifully. What we thought was happening: The adapter was ready to ship. What was actually happening: The first smoke test from a real client against the running server discovered that a required discovery feature had never been enabled, and the client had no way to find the service. Fix: two lines. The in-memory loopback tested logic; it did not test deployment. The retro's one-liner: "this is not an argument against unit tests; it is an argument for also just… running the thing." The lesson: §8 — your tests can only catch the things they actually test. A complete suite against the in-memory path is not a complete suite against the wire.

The Tests That Would Have Passed Anyway (case study 1)

What happened: A fix restored nginx routing so social-media crawlers could reach the link-preview endpoint. The change shipped with a suite of E2E tests. All green. What we thought was happening: The routing fix was verified end to end. What was actually happening: The tests hit the backend endpoint directly and the dev server directly — bypassing nginx, the exact layer being fixed. Reverted, every test would still have passed. A reviewer asked one question — would these tests pass if you reverted the change? — and the theater collapsed. The remedy was honesty in two moves: rename the test file to what it actually pins (the endpoint contract), and move the real verification into the post-deploy smoke test, the only place that exercises nginx. The lesson: §8 — the revert question is "the sharpest tool we have for distinguishing verification from verification theater."

The Formatter That Ate the Evidence (case study 1)

What happened: A consumer API-key tier shipped with forty green tests and a plan that ended with a mandatory manual integration task. The manual task found two production-killers in one afternoon. What we thought was happening: Forty for forty; the manual step was belt-and-suspenders. What was actually happening: First, the production compose file explicitly allowlists which env vars reach each container, and the new CONSUMER_API_KEYS wasn't on the list — the feature was "entirely correct in code review and entirely broken in production." Second, the logging pipeline's formatter had been silently discarding every structured extra={...} payload since the day it was configured; the test suite's log-capture fixture passed because it reads records before any handler runs. Neither failure was reachable by a unit test — one lived in an ops file no test loads, the other past the exact boundary the fixture stops at. The lesson: §8 — "when testing whether a feature delivers, not whether it runs, look at the wire output, not the test fixture."

The Perimeter That Trusted the Polite Client (case study 1)

What happened: Native mobile clients started getting 403s from endpoints that worked flawlessly in every browser and every test — and had for three major versions and 311 CI runs. What we thought was happening: A mobile-side auth bug; the captured response headers said X-Authenticated: true right next to the 403. What was actually happening: A "defense-in-depth" perimeter middleware had four authorization checks that were secretly sequential — three of them dead code — and the surviving check keyed on the Referer header, which browsers send on every XHR and the native HTTP stack never sends. Every test client was the polite kind. The fix was 21 lines: trust the request the upstream auth had already authenticated. The lesson: §8 — test with the rudest client you support. A guard that only ever meets polite clients is a guard nobody has met.

The Smoke Alarm the Security Release Disarmed (case study 1)

What happened: During an audit, someone noticed the frontend error tracker had reported nothing — nothing at all — for five releases. What we thought was happening: Five quiet releases. What was actually happening: A security release had tightened the Content-Security-Policy to connect-src 'self', which blocked the error tracker's own reporting endpoint. The release that hardened the app disarmed its smoke alarm, and nothing downstream of a dead instrument ever complains. The retro: "We have not yet decided whether this is hilarious or appalling." The lesson: §8 — observability is part of the system; verify it observes. Fire a deliberate test error through the pipe now and then.

Scope, plans, and surprises

Stories where the plan looked obvious, the scope felt small, and the thing that bit was the work nobody had named yet. These are the bruises that come from skipping the brainstorm, leaving a placeholder in a plan, or trusting a fan-out to hide coordination problems it never could.

Four Scoping Gaps in One Release (v5.11.0)

What happened: A one-line release-automation config fix was supposed to make Release PRs open against the develop branch and trigger the deploy workflow when a Release was published. The fix shipped. Nothing worked. Then the second fix shipped. Nothing worked. What we thought was happening: A typo in release-please-config.json — add target-branch, move on. What was actually happening: Four separate scoping errors, each plausible in isolation. target-branch is an action input, not a config key — the JSON accepted it silently. There was no anchor Release, so release-please scanned to the start of the repo. The merge commit used release: as its type, which is not a conventional-commit type and was silently skipped. And release-please plus the default GITHUB_TOKEN does not trigger downstream release: types: [published] workflows. The lesson: §14 — walk every spec section and name the task that implements it.

The naivePlural Drift (case study 4)

What happened: A small naivePlural() helper existed as three copy-pasted duplicates across three edge functions. Over one release cycle, one of the three copies had an irregular-noun branch extended with tooth → teeth and goose → geese; the other two didn't. The user-visible bug: the same entity rendered as "1 mouse" on one page and "1 mice" on another, with a third path still saying "1 mouses." What we thought was happening: A rendering inconsistency — maybe a stale cache on one of the pages. What was actually happening: Three copies of the same function that had drifted silently. Nobody owned the reconciliation because the copies were small enough individually that none of them flagged in review. The duplication was fine the day it shipped; what it became was a bug. The fix was to extract the helper into a shared module — the extract that, under §7's rule, had been deferred as "the pattern hasn't stabilized yet." It had. The stabilization was invisible because nobody re-checked. The lesson: §7 — duplicates are fine when the pattern is still finding its shape; duplicates that have already drifted are a bug waiting to be reported. Re-check at each correction, not just at the third.

The Four-Fix WCAG Contrast Cycle (v5.5.0–v5.6.0)

What happened: Across two releases, four separate WCAG contrast violations slipped into the UI — each one a utility-class color swap picked by eye that looked fine in the design tool, failed the axe-core suite on the next run, and got fixed with a one-class swap. What we thought was happening: We thought we were calibrated to the contrast rules by now. After the second one, we thought we were calibrated now. After the third, we stopped guessing. What was actually happening: Nothing was broken — the automated rail was catching exactly what it was designed to catch. The humans were the slow learners. The right move was to stop pre-judging contrast by eye and let the axe-core evidence do the approving. The lesson: §9 — when the blast radius is small and the test is automated, confirm by evidence, not by ceremony.

Hand-Rolling the Legacy State Machine — Right Call, Wrong Call (second case study, v0.3.0)

What happened: The second case study was implementing a batch of early-era protocols. One of them is the pathological one — a decades-old protocol that uses two connections for what every other protocol accomplishes with one, with a secondary data channel that requires hand-negotiated host/port encoding in a format nobody else uses. The team considered importing a library and decided instead to hand-roll the entire state machine from the standard library. ~150 lines. Pure stdlib. Listener, data channel, directory listing, file transfer. All of it. What we thought was happening: This is a judgment call about dependency weight versus implementation cost. The project's ethos was zero-dependency-if-possible; the protocol was in scope; the hand-roll was tractable; we'd save the dep. What was actually happening: The retro named the tension directly: "Right because it kept the dependency count low and forced us to truly understand the data-channel negotiation. Wrong because we now truly understand the data-channel negotiation, and that knowledge cannot be unlearned." The hand-roll worked. It shipped. The test suite ran clean. And the team now carries around the protocol's internal mechanics as permanent mental residue — the bespoke host-port encoding, the ephemeral listener, the two-connection dance, the state transitions that make sense only if you were in the room when the spec was written. Every unit of knowledge about a legacy protocol you gain is a unit of attention you cannot reclaim for anything else. The dependency you didn't import was free; the attention you spent internalizing the protocol is not. The lesson: A counter-weight to "always reach for a library" and to "always hand-roll for purity." The real question isn't library-vs-hand-roll — it's whether the knowledge you'll acquire during the hand-roll is worth the craft attention it will permanently occupy. Sometimes yes (the team's zero-dependency posture became a source of genuine craft pride across nine releases). Sometimes no (the legacy protocol's mechanics are nobody's art). The judgment call connects back to §1's three-layer thesis: a hand-roll is a decision to spend craft attention on mechanics, and that trade is only worth it when the mechanics are themselves part of the craft. Here they aren't. The zero-dependency posture as a whole is. The retro was honest about the tension instead of pretending the hand-roll was unambiguously correct, which is the only reason the lesson survives.

Fred Was Almost an Uncle (case study 5)

What happened: A subagent was dispatched to write reply templates in the voice of a beloved children's-television host. The draft came back warm, kind, encouraging — and generic. ("Yeah. That's a real one.") What we thought was happening: The dispatch was fine; the model just needed another pass. What was actually happening: The dispatch prompt said "the right energy" in passing and pointed at a JSON file — not at the twenty lines of seed library that actually specified the voice (the permission-granting constructions, the load-bearing second person). The human's review was one sentence: "these don't sound like fred rogers. Are you sure?" The re-dispatch put the voice spec in the prompt verbatim and required the agent to produce three too-saccharine and three just-right calibration samples before drafting. The second draft was the man himself: "You're a person who carried a worry without making a fuss about it… You did that." The lesson: §1 and §4 — for taste-heavy delegation, pointing at the spec is not the same as putting the spec in the context. Demand calibration samples before the real output, and route the result past the human ear.

The Streak in a Trenchcoat (case study 6)

What happened: During a pre-code elicitation session for an anti-optimization health app — a product whose ethos explicitly forbids gamification — the agent proposed a success metric: "you use it daily, unbroken." What we thought was happening: A reasonable north star for a daily-use product. What was actually happening: The metric was "a streak in a trenchcoat" — the exact engagement mechanic the product forbids, smuggled into the meta-layer by pattern-matching on what success metrics usually look like. The human's verdict on streaks ("Streaks stink of judgement") reversed the metric out of the specs entirely. The catch happened only because the elicitation session made the agent state its model of success out loud before any of it was load-bearing. The lesson: §4 — audit the agent's proposed north stars against the product's own ethos. The agent optimizes toward the pattern, and the pattern is somebody else's product.

Diagnosis & the lab

Stories where the fix was drafted before the evidence was read — and what happened when a project started shipping instruments instead of guesses. The bruises here come from trusting a diagnosis (the ticket's, the agent's, your own) that no trace had ever witnessed.

The Hedge That Wasn't (case study 5)

What happened: A ticket reported that hedge phrases ("just a little bit") were suppressing heavy-affect detection in a journaling classifier. A hedge-ignore subsystem was designed and half-drafted. What we thought was happening: Hedges were diluting the signal; the fix was to strip them before classification. What was actually happening: The persisted trace — newly available, because the previous release had shipped the diagnostic surface instead of a fix — showed the misfired entry had never matched anything: the word sad simply wasn't in the heavy-signals list, which spoke only in extremes (devastated, hollow, grief). Four lines of mild-affect vocabulary fixed it. "It would have been a beautiful fix for a problem that didn't exist." The lesson: §10 — the ticket's diagnosis is a hypothesis. Read the trace before believing the issue.

The Word the Domain Was Named After (case study 5)

What happened: A rest-and-recovery keyword domain kept missing obvious entries. What we thought was happening: Edge cases; the list needed a few more synonyms. What was actually happening: The agent-authored wordlist contained every verb of resting (napped, slept) and every resting-place noun (porch, hammock) — and not the word rest. "The bare noun. The thing the entire domain is named after. Missing… We added it. The fixture passed. We laughed." The sibling contentment cluster had the same disease: simple pleasures and at peace, no happy — vocabulary "designed by someone trying to write literature about contentment." The lesson: §1 — agent-authored heuristics encode the author's vocabulary, not the user's. Only real inputs close the gap.

The Marcus Monopoly (case study 5)

What happened: After a new reply register shipped, the user reported that one historical figure was answering every playground test. What we thought was happening: The rotation logic was broken; a fix to production fairness rules was taking shape. What was actually happening: The fairness rule — a figure not seen recently beats all others — is correct in production, where every reply becomes history. The debug playground loads history once and never writes back, so the one figure absent from the last sixty days won every single draw: a fairness rule producing perfect unfairness against frozen state. The user made the call: "I think the logic is right. I think we need to fix /debug so that it ignores the anti-repeat logic, since it will never save." Production shipped byte-identical; the playground got a bypass flag. The lesson: §10 — exploration mode is not production mode. When the bug only reproduces in the lab, suspect the lab's frozen state first.

The Silence That Was the Symptom (case study 1)

What happened: A pre-planning survey agent, sent to read the nginx layer before a routing release, mentioned almost as an aside: "nginx blocks social crawlers by default." What we thought was happening: A configuration note for the backlog. What was actually happening: Four curls confirmed that every shared link's preview had been a 401 JSON blob on every social platform for three releases — the link-preview generator built two versions earlier had never once been reachable from the surfaces it existed to serve, dropped in a later nginx consolidation. No one had filed a bug, because broken share previews don't generate bug reports; friends just don't click. "The silence is the symptom." The two-issue release became three, and the third was the one production needed most. The lesson: §5 and §10 — exploration finds the bugs nobody filed. Features whose failure mode is silence need scheduled probes, because no user will ever tell you.

Tooling & environment traps

Stories where the code was fine and the tool was lying. Bash buffering its own script, a token that couldn't wake the next workflow, a Slack dialect cosplaying as Markdown — the bruises here all come from trusting a tool to behave like its documentation said it did.

The Actions Token That Won't Wake the Next Workflow (v5.10.0)

What happened: We moved release creation to release-please and the deploy pipeline went silent. Merging the Release PR created the GitHub Release exactly as intended; the deploy.yml workflow, wired to release: types: [published], never fired. What we thought was happening: A YAML trigger typo, or maybe a branch-protection rule eating the event. What was actually happening: A documented GitHub Actions behavior: PRs and Releases created by a workflow using the default GITHUB_TOKEN do not trigger downstream workflows. The "fix" was technically working — a Release was created — but the next stage was built on a chain that the token could not carry across. A PAT or GitHub App token was required for the handoff. The lesson: §8 — verify in the environment that matters; a green release event doesn't prove the downstream workflow saw it.

Bash Buffers Scripts In Memory (v4.2.0)

What happened: The footer on the live site read rev unknown after deploy. The footer had displayed the git hash since v4.0.0. It worked in dev. It worked in CI. It did not work in production. We pushed fix after fix to deploy.sh and watched the old buggy version run to completion anyway. What we thought was happening: Git wasn't pulling, or the runner had a stale clone, or the GIT_REV export was being eaten somewhere in the compose plumbing. What was actually happening: Bash reads a script into a memory buffer when it starts executing. git pull updated the file on disk — the new bytes were right there — but the running shell kept executing its in-memory copy of the old file. Any edits after the git pull line never took effect on the run that pulled them. The fix is one line: after the pull, exec "$0" "$@" to re-read the script from disk. The lesson: §11 — when the obvious fix doesn't stick, stop editing and dump the raw state. (See DevOps Playbook Gotcha #9 for the exec "$0" "$@" self-reexec pattern.)

Slack mrkdwn Is Not Markdown (v5.10.0)

What happened: CI notifications started going out to the team channel with raw **asterisks** and [link text](urls) rendered as literal characters instead of formatting. The messages looked like a drunk bot. What we thought was happening: A webhook payload escaping bug — something double-encoding the Markdown before Slack got it. What was actually happening: Slack's mrkdwn is not Markdown. Bold is *single asterisks*, not **double**. Links are <url|text>, not [text](url). The webhook was delivering exactly what we'd written; we had written the wrong dialect. The fix was a full conversion to Block Kit, where the formatting contract is explicit instead of cosplaying as a familiar one. The lesson: §8 — verify in the environment that matters, including the one that renders your message. (See DevOps Playbook Gotcha #6 and Phase 6 for the Block Kit migration.)

The Truncated Secret (case study 6)

What happened: A serverless function kept rejecting a signing secret that had definitely, verifiably been set. The code that validated it came under suspicion. What we thought was happening: A validation bug, or a subtly wrong secret. What was actually happening: The platform CLI's env:get, used to source the secret into the session, returned a 20-character slice of a 64-character value — and exited zero. The function was correctly rejecting garbage; the system was working. The retro's warning: tooling can hand you a truncated value "and smile while it does it." The lesson: §2 and §9 — when a sourced secret is rejected, check its length before doubting the code. Verify properties of secrets (length, prefix), never by printing values.

The Day the Landlords Came Calling (case study 1)

What happened: In one release: the CI provider refused all five jobs — a billing dispute two vendors above the project — forcing a local fast-forward merge and a by-hand run of the deploy script. An hour later, the issue tracker's free-tier cap blocked filing the regression the release had just found. What we thought was happening: A CI outage; then a tracker hiccup. What was actually happening: Two SaaS dependencies failing on the same afternoon, both unrecoverable through the failed service. The deploy survived because the deploy script had always been runnable by hand. The bug survived because it got written to three tracker-independent places — the changelog, memory, and the retro. The aftermath is visible in the project's instructions file: the tracker was demoted, and the backlog went local-first. The lesson: §3 — every automated rail needs a rehearsed manual fallback, and every found bug needs a persistence path that doesn't depend on someone else's billing department.

The Redirects File That Outranked the Config (case study 6)

What happened: Wiring the third sibling app to the same API-endpoint pattern as the first two — "muscle memory" by now — produced 404s and SPA HTML from /api/v1/* routes that were configured identically to the working apps. What we thought was happening: The pattern was identical; something environmental had to be wrong. What was actually happening: The third app had a public/_redirects file whose bare catch-all outranks the netlify.toml redirects the pattern relied on. "The pattern we'd called 'identical' was identical in the code and different in the routing, and the difference lived in a file we hadn't thought to look at." The third repetition is when you stop reading the instructions — and when the platform's precedence rules stop forgiving you. The lesson: §5 — explore before you plan applies to config, too. "Identical to the last one" is a claim about every layer, including the ones you didn't diff.

Trust & blast radius

Stories about authorization, denial paths, and the moments when "yes" in one direction quietly failed to mean "yes" in every other direction. The bruises here come from forgetting that the interesting rules live in what gets refused, not in what gets allowed.

The Admin Who Couldn't Fire Herself (v4.3.0)

What happened: RBAC landed for the admin panel. The interesting bugs weren't in the "yes" paths — those were one-liners. They were in the "no" paths: self-role-change, self-deactivation, self-deletion, privilege escalation via update. What we thought was happening: RBAC is a matrix of allows; fill it in and you're done. What was actually happening: Trust in one direction ("you are an admin") does not generalize in every direction ("therefore you may fire yourself"). Half a dozen explicit refusals had to be written on purpose, one by one, because each was a line the matrix didn't draw by default. The last-superadmin protection came later still, in v5.6, because we missed it the first time too. The lesson: §9 — authorization is scoped, not blanket. Every yes is for one action.

The Default That Was Admin (case study 4)

What happened: A refactor of the edge-function pipeline introduced a shared handler signature with an optional client: parameter — 'admin' or 'anon' — that defaulted to admin when unspecified. Existing functions were fine because they passed the right client explicitly. Every new endpoint written afterward silently ran with admin privileges because not thinking about the client meant getting the admin one. What we thought was happening: The new signature was ergonomic; the existing callers were correct; the new callers were also correct because they compiled. What was actually happening: A security review in the next release caught two endpoints (a public stats endpoint, a public leaderboard) running with admin keys for a full release cycle. No data leaked because the underlying RLS happened to cover it, but the privilege gradient was inverted. Fix: one line — change the default to anon. The lesson: §9 — the unsafe default is the one that gets used by everyone who didn't think about it. Make the safe choice the default; require explicit opt-in for elevation.

The record that lied

Stories where the written context — memory, docs, comments, the harness itself — asserted something that was never true, and every reader repeated it with confidence. The bruises here come from treating written words as witnesses. Only a command output is a witness.

The Lock That Wasn't (case study 5)

What happened: The instructions file said main was branch-protected. The auto-memory said so. Weeks of PR descriptions said so. During an audit, someone finally ran the API call. What we thought was happening: Nothing — that's the point. The gate appeared to be doing its job. What was actually happening: 404: Branch not protected. The protection had never been enabled. "Both were lying. Not maliciously. Just confidently. For weeks." The gate had appeared to work because no one had tested whether the gate existed. The fix took two minutes; the reclassification took longer — the README now treats an un-PR'd commit on main as a security incident, and the belief carries its probe. The strange small joy from the retro: after enabling protection, "the auto-memory was not corrected. It is now correct without modification." The lesson: §6 and §15 — a belief you've never probed is a hypothesis. A protection you've never tested is a protection you don't have.

The Header Nobody Read (case study 1)

What happened: An nginx config comment confidently documented that the backend trusts an X-Forwarded-By header as part of the security perimeter. What we thought was happening: A load-bearing control, faithfully described. What was actually happening: grep -i x-forwarded-by backend/ returned zero matches. No middleware had ever read the header. The comment was born false and had survived every review since, because "a documentation lie about a security control … looks load-bearing in code review. It survives audits" — audits that read the docs instead of probing the claim. The lesson: §11 (failure mode seven) — written context is an amplifier, not a witness. Grep the claim, not the comment.

The Edit That Reported Success (case study 5)

What happened: Route components shipped to production that nothing routed to — pages that existed in the bundle and were unreachable in the app. What we thought was happening: The components had been wired; the harness said the edits succeeded. What was actually happening: The harness's edit tool had, at least once, reported success without persisting the change. The components compiled cleanly in isolation, CI stayed green, and the missing <Route> entries were invisible until a human clicked. The harness is software; software lies under load. The lesson: §2 — trust harness claims the way you trust "tests pass": with evidence. For edits that matter, the witness is git diff, not the tool's return status.

Parallel agents & architecture

Stories where the individual agents did clean work and the result broke anyway — because the shape of the collaboration had a hidden incompatibility with the tools underneath it, or because an effect emerged from the parallel structure that no single agent could see. Fix the topology, not the agents.

Parallel Work (case study 1)

What happened: Two agents dispatched to write tests in parallel — one in Python/pytest, one in TypeScript/vitest. Different directories, different build tools, no shared state. The backend agent finished in two minutes; the frontend took three. What we thought was happening: A risky parallel dispatch that we'd have to reconcile. What was actually happening: There was nothing to reconcile. Both agents produced working tests on the first run. The absence of conflict wasn't luck — it was what the file map guaranteed the moment the split was drawn. The lesson: §13 — fan out only when tasks are independent; group by file overlap, not issue priority. The file map is the contract.

The Subagent Orchestra (case study 1)

What happened: Fourteen subagents dispatched across two parallel tracks on a refactor. Twenty-nine files touched, zero merge conflicts on core logic. What we thought was happening: Fourteen is a lot of agents. We'd see conflicts. What was actually happening: The plan's file map — written before any agent started — assigned every file to exactly one dispatch. Conflicts weren't prevented by discipline; they were impossible by construction. The lesson: §13 — dispatch to compress process, not to avoid reading the result. Plan the file map before the fan-out.

The Wave Pattern (case study 2)

What happened: By the third release, the project had converged on a rhythm: parallelize in waves, not all at once. Trivial adapters (50 lines of stdlib) first. Moderate adapters (state machines, transport security) next. Complex adapters (hand-rolled state machines, multi-phase handshakes) last. What we thought was happening: A scheduling heuristic. What was actually happening: A load test for the harness. The trivial wave validated that the build system, config registration, and test harness all worked before the project committed agents to 150-line state machines. The retro: "the software equivalent of checking the parachute before jumping." Seven parallel agents per release, no coordination overhead, because the first wave shook out every infrastructure surprise before the expensive work started. The lesson: §13 — schedule the cheapest items first as a load test for the harness. The expensive work runs on a harness you've already debugged.

The Interface That Never Needed a Sixth Method (case study 2)

What happened: A five-method adapter interface — name, start, stop, port, protocol identifier — was designed in the first release and absorbed more than seventy adapters across nine releases without ever changing. TCP, UDP, TLS, HTTP, IPC, raw sockets, binary protocols, text protocols, XML streams, binary-encoded formats, sidecar-based adapters, platform-specific adapters. Five methods. Still five methods. What we thought was happening: A clean abstraction designed for extension. What was actually happening: Something stronger. The retro: "the interface isn't just a software abstraction — it's an organizational one. It turns a team coordination problem into a compilation step." When four agents work in parallel against the same contract, conflicts are impossible by construction — the interface refuses to permit them. The agents build in total isolation and merge without conflict because the only thing each adapter can do is satisfy the five methods. The lesson: §13 — the interface you design before the parallel work begins is the coordination protocol the parallel work runs on. Design it like it matters.

The v6 Worktree Experiment (case study 3)

What happened: Five parallel agents, five git worktrees, five feature branches, one codebase. Each agent's individual work was clean. The integration was a disaster. What we thought was happening: Worktrees were isolating the agents from each other. Git was handling the rest. What was actually happening: Worktrees isolate the filesystem; they do not isolate branches, refs, HEAD, or merge state. Agents committed to wrong branches, commits cross-contaminated between features, and orphaned refs had to be recovered by manually extracting files from abandoned commits. No agent was undisciplined. The coordination model had a hidden shared-state dimension the setup didn't anticipate. The retro: "parallel worktrees work best when agents are completely independent; shared git state (branch switching, merges) creates coordination problems." The lesson: §11 — when the agents are good and the result is bad, the architecture is wrong. §13 — worktrees are workspaces, not a coordination strategy.

The v7 Team Pivot (case study 3)

What happened: Same project, next release, same kind of parallel work. Instead of worktree-per-agent on separate branches, the team used a team-spawning mechanism with agents in the same worktree, on the same branch, with explicit file-boundary assignments before any work started. What we thought was happening: A last-ditch simplification. What was actually happening: Three agents on three separate features merged cleanly with zero conflicts. Two more agents ran in a follow-up phase with explicit file-boundary instructions for shared files like App.tsx and types.ts. Same project, same agents, same discipline as v6. The architecture changed; the outcome changed. The retro: "parallel agents work well when boundaries are drawn at the feature level (different files) rather than the branch level (different git state)." The lesson: §11 — you don't out-discipline a bad topology; you change it. §13 — isolate context with worktrees; isolate work with file boundaries.

The Vitest Cache Incident (v5.0.0)

What happened: Four parallel agents wrote theme variants across several pages. The code was clean, the tests were clean, the dispatch was textbook. Then thirty-eight unrelated tests started hanging with vitest timeouts, every one of them in files nobody had touched. What we thought was happening: An agent had subtly broken a shared util or introduced a render loop somewhere in a component mount path. What was actually happening: Four agents had been writing to the same node_modules directory at once, and vitest's module-graph cache under node_modules/.vitest had corrupted itself during the concurrent writes. No individual agent's output was wrong. The failure was an emergent property of the parallel structure. rm -rf node_modules/.vitest fixed all thirty-eight tests. The lesson: §13 — subagents protect your context window, they don't hide interactions between dispatched tasks from you.

The Phantom Downgrade (case study 5)

What happened: Days after a four-agent report-only audit fleet ran, the working tree showed a package.json version downgrade — 1.1.0 to 1.0.3 — that no human had made and no commit explained. What we thought was happening: Someone fat-fingered a revert. What was actually happening: Most plausibly, one of the "read-only" auditors had run a package-manager command mid-analysis, and the tool had helpfully rewritten the manifest in the shared tree. The audit itself was a clear win — eighteen issues filed in ten minutes — but "report-only" described the agents' assignment, not their side effects. Harmless this time; riding along in someone's next commit the time after. The lesson: §13 — isolate even the read-only agents. A fleet sharing your working tree is the topology failure in its mildest form, and the mild form is free to prevent.

The Docker Port Mappings That Weren't (case study 2)

What happened: 180+ passing tests. Zero data races. Four clean releases. And a Docker container that had been completely unreachable from outside for every single one of those releases. What we thought was happening: The container was fine — CI said so, and CI had been green on every dimension we'd thought to measure. What was actually happening: The Makefile's port mapping routed a host port to a privileged container port, and the binary inside the container ran as a non-root user that could not bind privileged ports and was listening on a different port entirely. Every port mapping was a Potemkin village pointing at an empty socket. The first manual smoke test returned an immediate TCP RST. CI had validated that the image built; CI had never validated that the image answered. Fix: fifteen lines across three files. The retro: "unit tests prove your code works. Integration tests prove your components work together. But nothing proves your deployment works except deploying it and poking it with a stick." The lesson: §8 — test what you ship, not what you build. Same scar as The Health Check That Wasn't, two projects, two languages, two different infrastructure stacks.

Audits as craft work

Stories where the human decided to look at the work, systematically, with no goal but to find what was wrong — and found things no amount of feature work, bug reports, or automated tests would have surfaced. These are the bruises that exist because nobody was auditing; the audits above healed more than any feature release could have.

The Security Audit Pass (case study 4)

What happened: A release was dedicated entirely to a systematic security audit — no feature work, no bug reports, no failing tests driving it. Just the human reading every endpoint, every query, every permission surface, looking for what a year of normal review had walked past. What we thought was happening: A nice-to-have. An hour spent sharpening the axe. What was actually happening: Fifteen issues surfaced, most of them category-level rather than line-level: lost-update races on entity mutation, TOCTOU on entity creation, missing UUID validation at API boundaries, regex-injection vectors in user-provided search terms, materialized views that had silently stopped being refreshed. None of these had triggered a bug report; most had been live for months; some had been live for over a year. Each one was found by slow, deliberate attention to a specific surface the normal review cycle never organized around. The lesson: Coda — the systematic audit is craft work the agent cannot substitute for and the pipeline cannot catch. Schedule audits on purpose.

The Visual Audit Pass (case study 4)

What happened: A later release walked every page of the app in both light and dark themes on both desktop and mobile, with no agenda. What we thought was happening: A polish pass. The design was already pretty good. What was actually happening: Six UX fixes surfaced that nothing else would have: inconsistent page-header sizes across sections, button overload on one page, featured content styled identically to error states, a confidence bar that took three rows to convey one piece of information, dark-mode contrast issues the automated axe suite wasn't flagging because they were at the threshold of "accessible but ugly." None of these were bugs. All of them were real. The audit produced more user-visible improvement than the preceding feature release. The lesson: §15 — audits are a different shape of attention than review. A reviewer checks that a change is correct; an auditor asks what's wrong with what's already shipped. The two disciplines produce different findings and are not substitutes.

The Claims That Ran Ahead of the Code (case study 5)

What happened: A "ready for company" release wrote the grown-up polish: a privacy policy, marketing copy, the works. Three patch releases over the following month quietly un-wrote it. What we thought was happening: The polish pass was the release's triumph — its retro celebrated the new copy. What was actually happening: The policy claimed paid-tier point-in-time backups (never purchased), claimed no third-party requests (a font CDN loaded on every page, leaking visitor IPs), and referenced an analytics provider (never installed). Aspirations, formatted as facts, published. The reversing retros stated the principle: "claims should be downstream of code, not upstream. Aspirational privacy text is a liability for exactly as long as it's unimplemented." And the companion norm: a retro that praised the mistake gets corrected out loud — "sometimes it's 'we got it wrong, here's what we did instead, and here's the principle.'" The lesson: §15 and §16 — publish what the system does; audit every outward claim like an assertion; and when a retro taught the wrong lesson, the correction is a retro-worthy lesson.

The Fleet in the Seams (case study 5)

What happened: Four parallel report-only agents — docs-vs-code, security, debt, devops — ran for about ten minutes and filed eighteen issues. What we thought was happening: A fishing expedition over a codebase that was, by every loop-level signal, healthy. What was actually happening: The codebase was healthy — "the debt was in the seams — between what we'd written down and what was true." The haul: the unprotected main branch everyone believed was protected, the backup tier that existed only in the privacy policy, a missing LICENSE, the IP-leaking font dependency, dependency drift. One-screen synthesis, eighteen tickets, one themed cleanup cycle. The next release's coda made the cadence argument: "The rot itemizes faster than it accumulates if you measure it at all." The lesson: §15 — the seams mechanize. A ten-minute agent fleet, run every few weeks, keeps the gap between record and reality enumerated — which is most of the way to keeping it closed.

---