Skip to content

Regex Frontmatter Parser Silently Swallows Block-Style YAML Lists

Problem

Several toolkit scripts parse spec frontmatter with regex instead of pyyaml — a deliberate, sanctioned choice (robust to the heterogeneous/malformed frontmatter in the corpus, zero deserialization risk). The idiomatic scalar getter looks like:

re.search(rf"^{re.escape(key)}:\s*(.+?)\s*$", frontmatter, re.MULTILINE)

The bug is the leading \s*. \s matches newlines, so for a block-style list —

depends_on:
  - SPEC-114

— the \s* after the colon crosses the line break and (.+?) captures - SPEC-114 from the next line. The getter returns the string "- SPEC-114" (a bogus scalar) instead of None. Downstream, a list parser that only understands flow style ([a, b]) then treats that as a one-element list and mangles or drops it.

Failure mode is silent: no exception, just a missing or wrong signal for the ~5% of specs that use block style. In SPEC-132 this dropped the depends_on structural signal that is the whole point of cross-project overlap detection (a false-negative in exactly the class the feature exists to catch). In the mirror shape-worthiness-score.py the identical bug went the other way — a block-style all-code deliverable_types read as ['- code'], a non-code false positive that inflated the shape score.

Pattern

Two-part fix, applied to every copy of the helper:

  1. Narrow the post-colon whitespace to [ \t] so it cannot cross a newline. A bare key: (block-style header) then yields None from the scalar getter, which is the correct signal that "the value is not on this line."
re.search(rf"^{re.escape(key)}:[ \t]*(.+?)\s*$", frontmatter, re.MULTILINE)
  1. Add an explicit block-style branch to the list parser: when the scalar getter returns None, capture the indented - item lines up to the next top-level key.
m = re.search(rf"^{re.escape(key)}:[ \t]*$(.*?)(?=^\S|\Z)",
              frontmatter, re.MULTILINE | re.DOTALL)

Guard callers that reason about emptiness (not just membership): a block-style list with items must read as non-empty. Route None (absent/block) vs a real scalar ([], null, ~) through different checks so null-literal handling is preserved.

When to apply

  • Any script parsing YAML frontmatter with regex rather than a YAML library.
  • Whenever a "value came back empty and I don't know why" bug touches a field that authors sometimes write in block style (depends_on, tags, deliverable_types).

When NOT to apply

  • If the script already uses yaml.safe_load — this class of bug can't occur. (But weigh that against the reasons the regex approach was chosen here.)

Mirrored-code corollary

The single most important lesson: when a regex helper is duplicated across scripts, its bugs are duplicated too. SPEC-132 fixed sibling_scan.py; the same split_frontmatter / scalar helpers lived verbatim in activate/scripts/shape-worthiness-score.py, which had to be fixed too. Before closing a parser fix, grep for the twin — and pair each parser with a block-style fixture so the gap can't silently reopen. (Same root as the newline-strict frontmatter delimiter fixed alongside this, review M2: CRLF / EOF-fence tolerance was also copy-pasted into both.)

Evidence

  • plugins/core-standards/skills/spec-from-requirements/scripts/sibling_scan.py (fm_scalar, fm_list, split_frontmatter) — SPEC-132 review H1 + M2.
  • plugins/core-standards/skills/activate/scripts/shape-worthiness-score.py (parse_frontmatter_scalar, parse_frontmatter_list, split_frontmatter, criterion (d)) — ported fix, same commit series.
  • Regression fixtures: test_sibling_scan.py::test_block_style_depends_on_surfaces, test_shape_worthiness.py::test_h1_block_style_* / ::test_m2_crlf_frontmatter_parsed.