Skip to content

A Probe That Does Not Fire Against the Unfixed Code Verifies Nothing

Problem

The natural order is: write the fix, write a test, see the test pass, ship. That order cannot distinguish a test that proves the fix from a test that would have passed anyway. Across four review passes on one file, five guards written this way were vacuous, and one bad mutation test nearly cleared a genuine, exploitable vulnerability.

The rule

Run the probe against the unfixed code first, and require it to FAIL. A probe is not evidence until it has been observed distinguishing the two states. Only then does its passing mean anything.

The cheapest reliable baseline is the code itself, not a hand-built mutant:

git show <pre-fix-ref>:path/to/file > /tmp/orig.sh   # then run the probe against BOTH

The five ways it goes wrong (each observed)

  1. The needle appears in its own source. An assertion grepped a function body for a config name; the function's own ⚠ comment mentioned that name, so it passed against a mutant with the code deleted. Grade behaviour, never source text.
  2. The mutation targets a site that does not decide. Removing a name from a precedence list changed nothing because a second discovery list still refused. Redundant defences make single-site mutation green — mutate the site that actually gates, or all of them.
  3. The test re-implements the code. A diagnostic was tested by an inline copy of the same loop, so reverting the shipped line left the test green. Extract the logic to ONE function and have both the production path and the test call it.
  4. The mutant is wrong, and the finding gets blamed. A mutation "cleared" a real fail-open, because the mutant reverted a delimiter while leaving a newly added fail-closed branch in place. When a mutation test clears a finding, suspect the mutant before the finding.
  5. The fixture cannot reach the failure path. A probe used the one input the code exempts, so it exercised the exemption instead of the reporting path it was written to test.

Corollaries

  • Use a capture-proof side effect. Write a file; do not print. Harnesses capture stdout/stderr, and a stderr probe reported zero executions while the payload was demonstrably running.
  • Pin the dangerous fixture, and say why. Where the exploit needs a specific hostile input (a filename that is also a glob, a config spelling that collides), a "tidier" fixture matches nothing and goes vacuous. Record in-file that the ugly name is load-bearing.
  • Label what you could not prove. Where behaviour is structurally guaranteed rather than guarded — e.g. pathlib making Path(base) / "/abs" equal /abs — the assertion is a property pin, not a mutation-proven guard. Saying so in the file is what keeps the next reader honest; an independent pass confirmed exactly that assertion reddened zero mutants, as labelled.

Relationship to existing patterns

Complements structural-test-assertions.md (assert the right location) and heuristic-calibration-gate.md (calibrate a detector's rate). This one is about the prior question: does the assertion discriminate at all?