A Redaction Self-Audit Must Detect a Superset of What It Strips

Pattern

When a sanitizer offers a self-audit mode — a --check that re-scans its own output and fails if anything sensitive survived — that detector must recognize a superset of every class the sanitizer strips. If the check is narrower than the strip rules, it returns "clean" on exactly the inputs the sanitizer failed to handle: the guardrail built to catch under-stripping is the thing that masks it. A false-clean self-audit is strictly worse than none, because it converts a silent leak into a certified silent leak.

Two invariants keep a self-audit honest:

1. Superset, not subset. Every strip rule needs a corresponding detector in the check. The safe drift direction is the check being broader (looser per-class thresholds), so a near-miss in a strip regex still trips the audit. Make this a maintained inventory (CHECK_PATTERNS[]) with a comment tying it to the strip rules, and seed one fixture per class so a future narrowing goes red.
2. Exclude your own placeholders. The audit scans sanitized text, which now contains the redaction tokens (NAME=[stripped], Bearer [REDACTED-CREDENTIAL]). A naive keyword detector re-flags token=[stripped] as a residual token=.... Strip the known placeholders before scanning — they can never appear in a real secret, so removing them cannot hide one.

This is the self-audit sibling of [[test-enforced-redaction-claims]] (seed the secret, assert its absence) and [[suppression-mirrors-detection]] (a suppressor must be anchored as tightly as the detection it cancels). Here the "suppressor" is the audit's own clean verdict: it must be at least as wide as the strip surface.

When this bites

The trap fires whenever all three hold:

1. A sanitizer has a --check/--verify path separate from its strip path.
2. The two share no source of truth — the check is a hand-written grep, the strip is a different engine (perl s///, a denylist, a library) — so they diverge on the next change.
3. Tests exercise the strip path but never assert the check itself fires on a residual (no teeth) or stays clean on already-redacted output (self-flag).

Example: SPEC-161 (SEC-2)

sanitize-capture.sh strips ~9 credential classes (Anthropic/OpenAI/GitHub/AWS/ Slack/Google/JWT/hex/bearer + base64 + keyword assignments) but its --check grep listed only sk-ant-|sk-|Bearer|/Users/|/home/|.internal…|://…@ — it omitted gh*, AKIA, xox, base64, lowercase-bearer, Google, hex, and every keyword assignment. So --check reported exit 0 "clean" on content still carrying, e.g., a ghp_… token or a client_secret= — the audit certified the very leaks the sanitizer (SEC-1) was under-stripping.

Fix: replace the single hand-rolled grep with a CHECK_PATTERNS[] inventory that is a documented superset of every strip class (looser thresholds — bearer ≥8 vs the strip's ≥20 — so a strip near-miss still trips the audit), and strip the redaction placeholders ([REDACTED-*], [stripped]) before scanning so the audit can't self-flag its own output. Regression tests give --check teeth: clean corpus → exit 0, a residual short bearer → exit 1, a fully-stripped credential → exit 0.

A companion gotcha surfaced in the SEC-3 fix: a perl s/OUTER/ … $v =~ m{INNER} … /ge replacement clobbers $1/$2 with the inner match's captures. Copy the outer groups into lexicals (my ($n,$v) = ($1,$2)) before running any inner m//, or the replacement rebuilds the string from empty captures.

How to apply

• If a redactor has a self-audit, make its detector an explicit, commented inventory and assert check ⊇ strip — grep the strip rules and confirm each has a check entry.
• Seed one fixture per class; add a teeth test that proves the audit can return non-zero, and a no-self-flag test on already-redacted output.
• Prefer the check being deliberately broader than the strip (more paranoid at build time) over risking it being narrower.
• Applies to any "verify our own output is clean" control: log scrubbers, PII redactors, secret masks, allowlist exporters.