Skip to content

Structural Test Assertions — Verify Location, Not Just Presence

Pattern

When a contract specifies where content must live (YAML frontmatter, a named section, a marker-delimited block, a nested config key), tests must verify the location, not just the presence. Substring assertions over a whole file (grep -qF "key:") prove only that the bytes exist somewhere — they cannot distinguish "correctly placed" from "placed in the wrong region but still in the file."

The fix is a small helper that extracts the target region first, then runs the membership check against the extracted slice. Pair every positive assertion with a negation assertion against the region's complement.

When this bites

The trap fires whenever:

  1. A loader/parser only reads a specific region (frontmatter, first JSON key, <head> block, etc.) for the contract field.
  2. The generator may emit the content outside that region (e.g., appended at end-of-file after the frontmatter delimiter).
  3. The test is a grep/includes()/assertContains against the whole file.

All three conditions held in SPEC-123 (see below) and the result was a Critical bug shipping through 12/12 passing tests for two consecutive review passes.

Example: SPEC-123

Context. SPEC-123 added regenerate-skill-paths.sh, which writes an AUTO-GENERATED paths: block into each SKILL.md. Claude Code's loader only reads paths: directives inside YAML frontmatter (between the --- delimiters at the top of the file). The whole point of the spec was to make product-scoped skills auto-load only in matching repos.

Bug. The first implementation of read_file_parts wrote the entire file body to one buffer and appended the new paths: block after it — i.e., outside the frontmatter, at end-of-file. The loader never saw it. The spec's primary mechanism (AD-5) was inoperative.

Why the tests didn't catch it. Tests T2, T11, and T12 used:

grep -qF "paths:" "$SKILL_MD"
grep -qF "AUTO-GENERATED by regenerate-skill-paths.sh" "$SKILL_MD"

These match the bytes wherever they appear — passes whether the block is in frontmatter or at end-of-file. All 12/12 SPEC-123 tests passed green for two consecutive review passes (Pass 1 and Pass 2). The bug was only surfaced when pattern-recognition-specialist (Pass 2) read the files manually and noticed the paths: block sat after the closing --- of the frontmatter.

Fix (Pass 2 → 60b3bd5c). Two complementary changes:

# tests/lib/frontmatter-assertions.sh
extract_frontmatter() {
  # awk between the first pair of '---' delimiters
  awk '/^---$/{c++; next} c==1' "$1"
}

assert_in_frontmatter() {
  local file="$1" needle="$2" msg="$3"
  extract_frontmatter "$file" | grep -qF "$needle" \
    || fail "$msg: '$needle' not in frontmatter of $file"
}

assert_not_in_frontmatter() {
  local file="$1" needle="$2" msg="$3"
  # body = whole file MINUS frontmatter
  awk '/^---$/{c++; if(c<=2) next} c>=2' "$file" \
    | grep -qF "$needle" \
    && fail "$msg: '$needle' should NOT appear outside frontmatter in $file"
  return 0
}

And regenerate-skill-paths.sh::read_file_parts was rewritten to re-emit frontmatter delimiters when reassembling the file, so the generated paths: block lands between description and the closing ---.

RED → GREEN proof. The new assertions FAILED against the old code, then PASSED after the fix. Without that proof the rewrite would have been "I assume this is better"; with it, the rewrite is verified to actually fix the unobserved bug class.

How to apply

For any contract of the form "X must live in region R":

  1. Write an extract helper. One function that pulls region R from the artifact. Use awk between delimiters for markdown/YAML, jq for JSON, an XPath-equivalent for HTML/XML. Centralize it — every test of this kind reuses the same extractor.

  2. Run positive assertions on the slice. extract_frontmatter "$file" | grep -qF "$needle" — never grep -qF "$needle" "$file".

  3. Pair with a negation assertion. "X must NOT appear outside R" catches misplacement bugs that the positive check alone misses — especially if a previous run left the content in the wrong place and the new run also emits it correctly (now you have it in both regions and the positive-only test still passes).

  4. Verify RED → GREEN. When introducing the helper, run the new assertions against the buggy code and confirm they fail. If they pass against known-bad code, the helper is also wrong.

Cross-applicability

This pattern recurs anywhere structural location matters:

  • YAML frontmatter fields in SKILL.md, spec.md, intake docs
  • AUTO-GENERATED blocks (paths:, symlink manifests, generated config sections) that must live inside a host file's metadata region
  • Markdown section content — "must appear in §3 Acceptance Criteria, not in §1 Problem"
  • JSON/TOML nested keys — config.section.field vs at root level
  • HTML attributes vs body text — <meta> head fields, ARIA roles, data attributes
  • Code-region contracts — "must be in the // AUTO-GEN START / // AUTO-GEN END block, not free-floating"

Reference

  • SPEC-123: Skill Scope Manifest — repo vs global vs product
  • Pass 2 fix commit: 60b3bd5c
  • Helper: tests/lib/frontmatter-assertions.sh
  • Review-history recurring-finding row: "Test asserts substring-on-whole-file instead of structural location" (first seen 2026-05-13, pattern-recognition-specialist)