Skip to content

A Baseline Written on macOS Can Go Red on Linux — Case-Insensitive File Lookup

Problem

SPEC-212's spec-schema-check reads spec.md and state.yaml from each spec directory, and guard 126 compares its findings with a committed shrink-only baseline. The baseline was generated on macOS and passed every local run. On PR #454, the ubuntu plugin-suites job reported one violation that the baseline didn't have.

specs/140-review-to-task-pipeline/ contains SPEC.md in upper case and no spec.md:

Platform lstat("…/spec.md") Checker verdict
macOS (APFS, case-insensitive by default) finds SPEC.md missing-frontmatter (that file has none), which the baseline recorded
Linux (case-sensitive) FileNotFoundError missing-file, a row the baseline did not have

Both answers are internally consistent. The defect is that the checker's verdict depended on the filesystem, so a baseline written on one platform could not hold on the other.

Root cause

os.lstat, open and os.path.exists all go through the filesystem's own name lookup. On a case-insensitive volume, a lookup of spec.md succeeds for SPEC.md, and nothing in the result tells you the stored name differs. Any check shaped like "does file X exist / read file X" is therefore platform-dependent whenever the tree can hold a differently-cased name.

Resolution

Require the exact name in the directory listing before reading:

if os.path.basename(path) not in os.listdir(os.path.dirname(path) or "."):
    raise Unreadable("missing-file", "absent (only a differently-cased name exists)")

Now both platforms report missing-file for spec 140, which is the true state of the repository. The one baseline row changed from missing-frontmatter to missing-file.

The test that passed for the wrong reason

The first regression fixture was:

printf '# no frontmatter\n' > "$R/specs/1-clean/SPEC.md"; rm "$R/specs/1-clean/spec.md"

On macOS, writing SPEC.md overwrote the existing spec.md (they are the same file), and the rm then deleted it. So the directory held no file at all, the unfixed checker reported missing-file, and the test was green before the fix. Deleting first, then writing, gives the directory that actually reproduces the bug:

rm "$R/specs/1-clean/spec.md"; printf '# no frontmatter\n' > "$R/specs/1-clean/SPEC.md"

With that order, T39/T39b/T39c in plugins/vt-base/tests/test-spec-schema-cli.sh are red on the unfixed checker and green on the fixed one.

Prevention

  • Any checker that decides "file present" must compare exact names, not rely on a successful open/lstat, if its output is committed or compared across machines.
  • A fixture that creates a case variant must remove the original first. Otherwise, on macOS, it silently tests an empty directory. Confirm it goes red on the unfixed code before trusting it.
  • Show the row, not just the count. The B3 self-test filtered the guard's output to lines matching B3|FAIL, which dropped the indented row. It now also prints the guard's indented detail lines, so the next CI failure of this kind names its cause.
  • A baseline generated on one OS is only evidence for that OS until CI on the other OS has passed.
  • docs/solutions/patterns/structural-test-assertions.md: assert behaviour, and prove the test can fail.
  • .claude/rules/testing-rules.md: "Assert that the fixture REACHED the branch".