Skip to content

A Security Fixture Must Prove It Reached the Branch Before It Asserts Anything

Problem

During one review of SPEC-203, five freshly written security assertions passed without being able to fail. Not because the assertions were wrong — each stated the right thing — but because the fixture never entered the code path under test. The assertion then held for a reason unrelated to the guard it was supposed to defend.

The five, with their causes:

Case Why it could not fail
(a2) v1 placed above where its bash helpers were defined; calls errored, $OUT was empty, and a "payload absent" check passed on an empty string
(a2) v2 the hostile entry was refused for an unrelated reason, so the exploit path never ran
(a3) the "no forged line" row was defended by a second lock, so removing the first changed nothing
(m3) the payload contained /, so mkdir -p built a nested tree; the entry landed at a depth the store glob never matches, self_entry returned None, and all four rows passed on a branch never entered
(i2) family both halves of a two-lock guard survived mutation individually, because no input distinguished them

Every one was caught by a surviving mutation, never by reading the test. Three of them looked perfect in review.

Root cause

A passing security test has two independent prerequisites, and only one of them is visible:

  1. the code does the right thing — what the assertion states;
  2. the fixture actually reaches the code — what nothing states.

When (2) fails, (1) is never exercised and the test reports success. Green is therefore ambiguous: it means "correct" or "never ran", and the two are indistinguishable from the output. Security fixtures are unusually prone to this because their setup is adversarial and fiddly — hostile filenames, planted symlinks, chmod 000, control bytes — so there are many more ways for the fixture to miss than for a normal one.

Resolution

Make reachability an assertion, not an assumption.

Before asserting anything about behaviour, assert that the branch under test was entered:

# Prove the fixture reaches the branch under test before asserting anything about it.
SELF_ACC=$(... resolve what the production code will resolve ...)
[ "$SELF_ACC" = "$EVIL_ACC_FIRST_LINE" ]
check "(m3) fixture reaches the command branch: self entry is the hostile account" $?

and, where a guard only fires if the unguarded path would have succeeded, construct that precondition explicitly and assert it:

mkdir -p "$R/outside/victim"        # the escape target must EXIST …
[ -d "$R/outside/victim" ]          # … or isdir would refuse for an unrelated reason
check "(i2) fixture is reachable: the escape target exists, so isdir alone would allow it" $?

Three further rules that fell out of the same review:

  • Mutate every new guard, and read the count. A guard whose mutation kills fewer rows than expected is the signal. Two of the five above were found exactly that way.
  • For layered guards, find an input each layer catches alone. Otherwise both survive mutation individually and the layering claim is unproven. Here: a uuid-shaped name that symlinks out of the store (only containment catches it) versus . which resolves inside but names nothing (only the shape catches it).
  • Keep the must-allow row beside the must-deny row, and make sure it can distinguish. A control asserting "not refused" is useless when the input is refused for another reason; assert "not refused by this guard".

Prevention

  • Treat a new security case as unproven until a mutation has killed it. First-run green is not evidence — it is the state a vacuous test is also in.
  • Watch for fixtures whose setup can silently relocate things: / in a name that becomes a path, mkdir -p on untrusted strings, chmod that makes a later step skip, helpers used before definition.
  • When a mutation survives, do not assume the mutation was wrong. Three of the five here were found because a survival was investigated rather than explained away.
  • docs/solutions/security/enumerate-untrusted-surfaces-not-fields.md — the fixes these tests were meant to defend.
  • .claude/rules/testing-rules.md — carries "a test that cannot fail is worse than no test" and mandates mutation. This pattern is the missing operational half: mutation tells you that a test is vacuous, a reachability probe tells you before you rely on it.