Skip to content

Revert-Proof the Negative Assertion — Seeding Is Not Reaching

Pattern

test-enforced-redaction-claims.md requires that a "we never emit X" guarantee be backed by a test that seeds a fixture containing X and asserts X is absent. That rule is correct and necessary. It is not sufficient.

A seeded negative assertion is only evidence if the seeded input reaches the code path under test. When it does not, the assertion passes for a reason that has nothing to do with the guarantee — and it passes just as happily with the fix removed.

The rule: for every negative assertion, revert the fix and confirm the test goes RED. Record that you did. Until then the test is a hypothesis, not a guard.

Why seeding fails to reach

Three ways, all observed in a single test suite (SPEC-167):

Failure Seed Why it never reached the assertion
The failure was swallowed PATH shim planting failing npx/git stubs; assertion = "the render still succeeds" The stub exited non-zero and wrote to stderr. The code called it as x="$(git … 2>/dev/null)", which absorbed both. The render succeeded because the call was ignorable, so the assertion held while the script really did spawn git.
The assertion matched prose, not code ! grep -qE '(npx\|curl)' "$HELPER", paired with grep -q 'grep' "$HELPER" to prove non-vacuity The pair matched the file's own header comment. The script never invokes grep at all. The "proof of non-vacuity" was itself vacuous.
The hostile input resolved somewhere harmless .active-spec = ../../etc/specs/1; assertion = "no .. or / in the output" Un-sanitised, the traversal resolved to a directory the fixture never created. The lookup missed, the fallback branch rendered a generic line, and the assertion held over a wide-open hole.

All three shared one shape: an absence was asserted without proving the presence was reachable. All three were written alongside a real, reproduced defect — the author had a working exploit in hand — and all three still failed to guard it.

The check

After writing a negative assertion, before trusting it:

cp target.sh /tmp/target.bak
# revert ONLY the fix the assertion guards
<apply the inverse edit>
bats suite.bats -f "<the assertion>"     # MUST be RED
cp /tmp/target.bak target.sh
bats suite.bats -f "<the assertion>"     # MUST be GREEN again

If step 2 is green, the test is not evidence. Fix the test, not the code.

Make the assertion depend on the mechanism, not on a side effect:

  • Swallowed failure → have the stub leave a sentinel the caller cannot redirect (append to a file), and assert the file's absence. Then arm the tripwire in the same test to prove it fires.
  • Prose match → strip comments before grepping (grep -v '^[[:space:]]*#'), and pin a command the code genuinely runs.
  • Harmless resolution → plant the target at the hostile destination, so unsanitised input really resolves it, and assert on the resolution ([[ "$out" != *"done"* ]]) rather than on characters in the output.

When this bites

  • Security fixes, where the negative is the guarantee and a false green is worse than no test.
  • Any "we never call / never emit / never read" claim.
  • Guards written from a reproduced exploit — the exploit is in the author's head, so the test looks obviously correct and gets less scrutiny than an ordinary assertion.

Relationship to the sibling patterns

  • test-enforced-redaction-claims.md — seed the input. Necessary; this doc adds and prove it arrives.
  • structural-test-assertions.md — extract the region first, then assert membership. Both are cases of the same root error: asserting over a scope you have not established.
  • self-referential-artifact-tests.md — a test whose subject can change underneath it.

Evidence

SPEC-167, one suite, three vacuous guards, none found by reading:

  • 1b940e52 — PATH shim replaced with a sentinel file after the swallowed-failure mutant passed.
  • 55d4a042 — grep -q 'grep' corrected after it was found to match only a comment.
  • c80dfa33 — traversal guard armed by planting the target at the traversal destination; reverting the sanitiser now prints traversal resolved out-of-root: ../../../../etc/specs/1 · done.

The third was found by an external security review, not by the author — after the author had already been caught by the same shape twice. That is the argument for making the revert-proof a mechanical step rather than a matter of attention.