Skip to content

A Fixture's Necessary Preconditions Define a Region It Can Never See

Problem

A test fixture often has to establish a precondition before the behaviour under test is observable at all. That precondition is not a shortcut and not laziness — without it the assertions would be vacuous. But it silently partitions the input space, and the complement of the precondition becomes a region the fixture is structurally incapable of reporting on. Assertion count, mutation-proofing and negation pairing all say nothing about it, because they measure the quality of assertions inside the observable region.

The failure is not a red test or a vacuous test. It is a green test next to a false claim, where the claim happens to live in the unobservable region.

Worked example (SPEC-177)

scripts/tests/setup/59-shipped-hooks-is-one-deliberate-entry.sh seeds every scenario's hooks dict with "_managed_by": "company-claude-toolkit". Its own header states why, as seeding fact 2:

Without the marker, merge() step 2's user-wins rule skips the toolkit value entirely and --safe is a NO-OP on hooks — rows 3/4/8 would then be testing the seed, not the installer.

That reasoning is correct. An unmarked seed would make the rows vacuous. The guard runs 27 assertions, pairs every positive with a negation, and carries mutation evidence for the rows that were green on arrival. It is one of the stronger fixtures in the repo.

It is also, by construction, blind to unmarked keys — and a hand-maintained ~/.claude/settings.json typically is unmarked.

The review then produced remediation advice for existing machines: "removing it is setup.sh --safe." That advice was wrong in exactly the unobservable region. On an unmarked key, merge_settings.py's user-wins rule makes --safe skip hooks entirely, so the command completes successfully and removes nothing. Published, it would have been advice that appears to work and silently does not — strictly worse than no advice.

Nothing in the suite could have flagged it. What flagged it was one command against a real artifact:

python3 -c "import json;print('_managed_by' in json.load(open('$HOME/.claude/settings.json'))['hooks'])"
# False

Why the usual quality signals miss it

Signal What it measures Why it is silent here
Assertion count gate that no block stopped running all 27 ran, inside the observable region
Mutation proof that a row can go red it can — for marked inputs
Positive/negative pairing that absence is not the only claim correctly paired, same region
Coverage of the diff that changed lines are exercised they are

Every signal is about assertion quality. None is about input-space reachability.

Prevention

  1. Name the precondition as a declared blind spot, next to the fixture. If the setup comment already explains why the precondition is required — as guard 59's does — it is one sentence further to say what that makes unobservable. The reasoning is already written; only the consequence is missing.
  2. Never source user-facing remediation advice from the fixture's assumptions. A claim about what a command does on a user's machine must be verified against a user's machine, or against the code path for the complement of the fixture's precondition — never against the suite's green result.
  3. When advice concerns a mode the fixture had to normalise to test, treat it as unverified until probed. The tell is the setup comment itself: any sentence of the form "we must seed X or the rows are vacuous" marks ¬X as unexplored territory.
  4. Do not "fix" this by dropping the precondition. That trades a blind spot for a vacuous suite, which is worse. Add a probe or a separate row for the complement if it matters — SPEC-177 chose to document rather than extend, because the complement's behaviour (--safe is inert) is correct and only the advice was wrong.
  • structural-test-assertions.md — assertion shape inside the observable region
  • suppression-mirrors-detection.md — the adjacent failure, where a suppressor is anchored more loosely than its detection and flips false-positives into false-negatives
  • probe-must-reproduce-against-unfixed-code.md — the same "verify against reality, not against your own scaffolding" instinct, applied to bug reproduction
  • handoff-notes-are-not-evidence (memory) — prose claims fail on disk as often as they hold