Skip to content

Fix-Commentary Drift

The failure

Fixing a review finding produces two artifacts: a code change, and prose explaining it. The code change gets tested. The prose gets nothing.

This asymmetry is invisible during the fix because writing the explanation feels like the verification step — you restate the reasoning, it reads coherently, you move on. But a sentence like "the real case is already covered by the sibling marker" is a factual claim about the codebase, exactly as checkable as an assertion, and nothing checks it.

In SPEC-156, five of six Pass-2 High findings were this one defect in a single pair of fix commits:

Prose added Reality
"already covered by plugins/vms/" That marker yields zero on this manifest — and the same ledger entry said so two bullets earlier
A corrected −6/+2/+1 decomposition Reconciled to 23 or 21, never the stated 22 — inside the paragraph written to fix an unreconciled sum
A comment explaining a deleted test block Named the wrong tests as covering the deleted guarantee
spec.md FR-2, plan.md marker table Never amended; the requirement still described the pre-fix predicate in both directions

The arithmetic case is the sharpest: the fix for "this decomposition does not reconcile" was a new decomposition that also did not reconcile, because an explicitly approximate source figure (~6 (see caveat)) was promoted to an exact subtrahend. The estimate was the error, and restating it preserved the error.

Why review does not catch it

Reviewers score claims against claims. Pass 1 scored a predicate finding against what the change ledger asserted, not against what the requirement said. Both were prose; they agreed with each other; the requirement was wrong. The DEVIATED requirement survived a full six-reviewer pass and was found only when a later pass diffed the shipped predicate against spec.md directly.

The fix commit is the least-reviewed commit. It arrives after the verdict, framed as closing findings rather than introducing them, and the reviewer's attention is on whether the original finding is gone.

The pattern

Treat every factual claim in fix prose as an assertion, and run it.

Concretely, before committing a review fix:

  1. Grep the claim. "Covered by X", "already handled by Y", "this only affects Z" — each is one command. In SPEC-156 every false claim was a single grep -c away from being caught.
  2. Re-sum every arithmetic. If a decomposition appears in prose, add it up. If a term came from an estimate, either measure it or keep it labelled as an estimate — never silently promote ~6 to 6.
  3. Follow the fix upstream to the requirement. A predicate change that is not reflected in spec.md / plan.md leaves the requirement formally deviated, however well the code works. Amend with a dated note rather than a silent edit, so the change is legible as a decision.
  4. Prefer a pointer to a restatement. Measured figures belong in one place (the decision record); code comments should carry the invariant and cite it. Restating a measurement in three surfaces creates three things to keep true.
  5. Ask what the comment claims that the tests do not. A comment asserting "every bucket is pinned by name in T7/T8/T9" is checkable — and in SPEC-156 it was false for two of four buckets.

The recursion is the point

While documenting this pattern, the same defect was committed again: a reviewer's "marker count is unchanged at 13" was propagated into three surfaces without counting the array, which holds 12. (The reviewer's own report contradicted itself two paragraphs later with "position 4 of 12".)

It was caught by a deliberate claim-check pass over the fix prose — the practice this doc describes — which is the honest evidence that the practice is needed and that intent alone does not substitute for it.

Two habits follow:

  • Record what you could not verify. SPEC-156's state.yaml carries a not_verifiable_locally list (GNU-grep behaviour on a ugrep dev machine; process-count figures attributed to a reviewer). An unverifiable claim is fine; an unverifiable claim presented as verified is not.
  • Distrust inherited numbers. A figure from a reviewer, a commit message, or a prior ledger entry is a claim, not a measurement. SPEC-156 carried a test count of 49 through three documents; it had never been executed.
  • count-reconciliation-single-source.md — the same disease in counts: reconcile to one authoritative source rather than restating.
  • three-pass-spec-validation.md — different passes catch different fault classes; this is the fault class that survives all of them by being prose.
  • self-referential-artifact-tests.md — the code-side sibling from the same spec: an assertion with no path to red.