Skip to content

Measure a Hidden-Character Filter Against the Whole Code-Point Space

Problem

claude-boot status reads the text of other Terminal windows and prints it in a report that an agent relays to the user. Window text is untrusted, so the shared flattener (plugins/vt-toolkit/skills/session-restore/scripts/lib/sanitize.py) must replace every code point that a terminal acts on or that renders as nothing. Otherwise text that the reader cannot see rides along in a relayed line.

Three review passes each found a new class the filter missed:

Pass What passed the filter Why
1 Unicode format characters (zero-width space, bidi overrides) the filter listed C0/C1 controls only
2 the variation selectors and 43 other format characters the fix was a hand-kept list of ranges
3 3,744 more code points the second fix decided by category (Cc, Cf, Zl, Zp, Cs), but many default-ignorable points sit in other categories: Hangul fillers, Khmer inherent vowels, the reserved U+E0000–E0FFF block

Every fix was correct for the examples in its finding. The filter kept leaking because its test checked those examples, not the property the filter is meant to implement.

Root cause

The filter's contract was "everything that renders as nothing". Unicode publishes that set as the Default_Ignorable_Code_Point property in DerivedCoreProperties.txt. The implementation approximated the property, with ranges and then with categories, and the test checked a sample. A sample cannot show where an approximation ends.

Resolution

  1. Name the property the filter implements. Here that is: categories Cc, Cf, Zl, Zp, Cs, plus every Default_Ignorable_Code_Point, plus the braille blank U+2800.
  2. Test against the property itself, not a sample. To find the gaps, pass 2 ran the filter over all 0x110000 code points. The test that stays in the occupancy suite (plugins/vt-toolkit/tests/test-spec-203-occupancy.sh, SAN_PROBE) walks every point of the default-ignorable set and asserts ignorable-missed=0. It also checks each range's visible neighbours and ordinary text (é, CJK, emoji) and asserts eaten=none, so a filter that hides too much fails as well.
  3. Mutation-check the boundaries. Shift one range end by one code point and the probe must go red. A same-size mutant can survive in a stale __pycache__, so delete it around every mutant.
  4. Say what the filter costs. An emoji drawn with a variation selector or a zero-width joiner now prints its parts with ? between them. The CHANGELOG states this.

Prevention

  • For any allow-list or deny-list over a character set, write the test as an enumeration of the whole input space against the published property. Never write it as a list of known-bad samples.
  • When a review finding names one character, re-run the enumeration before fixing. The finding is one point of a set, and the set is what the next pass will find.
  • Related: enumerate-untrusted-surfaces-not-fields.md, SPEC-203's lesson for the same module, one level up: enumerate where the bytes come from, then guard by construction.