A type-changing fix can make a sibling guard's test vacuous
Problem¶
SPEC-179's detector reads package.json with json.loads. Review pass 1 added a guard,
isinstance(body, str), so a non-string scripts.test value cannot raise TypeError, and pinned it
with a row using {"test": 5} — red when the guard was removed.
Review pass 2 fixed an unrelated CPU problem (a ~1 MiB integer literal parsed in 8 s) with
json.loads(raw, parse_int=str, parse_float=str, parse_constant=str). That fix was test-first too,
and correct. But it changed the type every number arrives as: 5 now reaches the rule as the
string "5". The pass-1 row kept passing — it still asserted npm test — while never reaching the
isinstance guard again. Deleting the guard left the suite green.
Nothing about the pass-2 change touched the guard, its row or its function's control flow. Reading the diff would not have found it.
Solution¶
Re-run the whole mutant set from earlier passes after every fix, not only the mutants for the
finding being fixed. The pass-2 run killed all 11 new mutants and then reported one survivor from
pass 1: SIMP-4 drop isinstance str. The row moved to JSON true and null, which still arrive as
non-strings, and the mutant was seen red again.
Keep the mutant list as a file (a small script that applies each old → new replacement to a backed-up
copy, runs the suite, records killed/survived, and restores), so re-running it is one command.
Prevention¶
- A fix that changes the type, shape or encoding of data at an input boundary (a parser hook, a decode step, a normaliser) re-routes every downstream branch that dispatches on type. Treat it as touching every guard behind it.
- A row's reachability is a property of the whole pipeline in front of it, not of the row. Assert it
where you can (
assert isinstance(parsed_value, int)in a fixture would have redded at once). - Expect the old mutant list to go partly stale: a mutant whose target text the fix replaced reports "0 occurrences". That is information too — retire or re-target it, never delete it unread.
Related: docs/solutions/patterns/security-fixtures-must-prove-reachability.md.