A Verdict Without a Denominator Reads the Same for N and for Zero

Problem

A verification step reports success without reporting its scope. PASS (all checksums verified) is true when 26 files were checked and equally true when zero were, because the sentence never names a number. Every bypass of such a check works the same way: don't defeat the comparison, remove the thing from the set being compared.

In BUG-077 the hook-integrity verifier had been silently checking nothing in the toolkit repo for 144 days, while printing a full-verification claim on every run. Nine pins had drifted. The check was not broken — it was unreachable, and its verdict could not express the difference.

The three failures, in order of subtlety

1. No denominator at all. PASS (all checksums verified) — an empty input loop produces the same line as a complete one.

2. A denominator with no independent anchor. Printing checked=26 is necessary and not sufficient. If the expected count is derived from the same list being checked, deleting an entry lowers both in lockstep:

whole entry deleted + backdoored hook  ->  checked=25  mismatching=0  PASS  exit 0

The file is counted in none of the outcome buckets — not checked, not mismatching, not missing, not unparsed. It simply left the population, and the accounting stayed internally consistent.

3. No closure. Even an anchored count only covers what the list claims. It cannot see a file that was never listed. A checksum allow-list is only an allow-list if every executable in an approved location must appear in it — the filesystem supplies the independent side of the comparison that the list cannot supply about itself.

Solution

1. Every verdict prints its denominator, and every outcome bucket alongside it: checked=27 mismatching=0 missing=0 unparsed=0 unpinned=0. Keep the buckets separate — each names a different attack with a different remediation, and collapsing them recreates the undifferentiated verdict.
2. A zero denominator is a finding, never a pass. "I examined nothing" is a failure to verify, not a successful verification.
3. Anchor the expected count outside the source under test. If expectation and actual come from the same file, the pair proves only self-consistency.
4. Close the set. Enumerate the population independently (the filesystem, a registration surface) and require every member to be listed. Match on the property — "a thing this repo executes" — not on a naming convention.
5. Tests must assert which finding, not that some finding occurred. See below; this is the failure mode that survives all of the above.

The trap: fixes that carry the bug into the new code

Across one branch this defect was reintroduced three times, by the fixes for itself:

the fix	the bug it carried in
strip comments, to rescue a trailing-comment key	a leading-comment key is now erased, deleting that entry
pin the count to the manifest	derived comment-blind, while the code strips comments first
add a closure scan	globbed *.sh, so the one extensionless pinned file stayed exploitable

Each fix reasoned about the case in front of it rather than the property. When you close one instance, state the property out loud and check the new code against it — not against the instance you just fixed.

The trap that outlives the fix: tests that pass for the wrong reason

Adding the closure scan made two existing regression cases stop testing what they named. Their assertion was "a CRITICAL appeared and the exit code was non-zero" — and closure now supplies both, for an unrelated reason. Mutating the parser to reintroduce the original bypass left the suite green:

mutation: key regex -> quoted-only   (this IS the original bypass; parser reads ZERO entries)
result:   guard PASSES — closure raised 27 unrelated CRITICALs and the case accepted them

The fix is to make each case assert its own signature, not merely that something complained:

• entry still read → checked stays at the full count, finding is a checksum/unparsed one
• entry genuinely removed → checked drops by one, finding is unpinned

A case that drifts onto the other mechanism then fails instead of passing quietly. A test can be green, non-vacuous, and still not testing its stated property.

Verification

Mutation-test each guarantee, and require the failure to name the case that owns it. Two confounds to design around:

• Self-reference. If the verifier is itself in the list it verifies, mutating it invalidates its own entry, so the run goes red for the wrong reason and hides the real verdict. Re-pin the mutated copy before reading the result.
• Over-loose matchers. Loosening assertions away from exact prose is right (see structural-test-assertions.md), but it has a floor: a case-insensitive match on critical also matches the summary line RESULT: 0 critical, 0 warnings. Anchor finding matchers to the finding format.

Where this keeps reappearing

Independently rediscovered six times in this repo before it was written down — twice inside the work that fixed it:

• BUG-066 — a working detector decayed into a no-op
• 54-spec-status-is-declared-and-agrees.sh — skip count printed every run
• 57-plugin-component-counts-agree.sh — skipped-plugin count printed every run
• 58-deliverable-types-are-declared.sh — key accounting balanced with -ne 0, not -gt 0
• repo-audit-ci.sh Check 0 — the checked=… accounting line
• guard 60-hook-integrity-cannot-fail-open.sh — its own first denominator was satisfiable by 1 of 26

The last row is the strongest argument for the pattern: the defect reproduced inside the guard written to prevent it, by an author who had spent a day on exactly this.

Related

• suppression-mirrors-detection.md — a suppressor must be anchored as tightly as the detection it cancels; the denominator case is the same rule applied to a counter
• structural-test-assertions.md — extract the region, assert membership, pair each positive with a negation
• test-enforced-redaction-claims.md — a "we strip X" guarantee needs a fixture that seeds X
• count-reconciliation-single-source.md — reconcile counts to one source; pin all N mirrors