Heuristic Calibration as a Hard Delivery Gate

Problem

Pattern-matching heuristics in workflow gates can fire on well-formed inputs, eroding user trust in the gate over time. If the false-positive rate is not validated before shipping, the gate becomes noise — users learn to dismiss it.

Discovered when building gray-area-heuristic.py for SPEC-127: the calibration task was initially advisory ("run and note the results"), but the false-positive risk warranted elevating it to a ship-blocker.

Solution

Include a mandatory calibration task in plan.md that:

1. Runs the heuristic against a real corpus (5–10 existing spec/PRD files that are known-good — i.e., already shaped or clearly well-formed)
2. Records the false-positive count and rate
3. Requires that rate to be acceptable before the spec can be marked complete

The calibration task should be listed in tasks.md as type: gate — not type: task — so it cannot be skipped without an explicit gate bypass.

Implementation Example (SPEC-127)

# tasks.md excerpt
- id: T11
  title: Calibration gate — 0 FP on well-formed corpus
  type: gate
  acceptance: |
    Run gray-area-heuristic.py against specs/[N]-*/spec.md files with
    status=completed. Verify 0 false positives (no gray-area prompts
    for well-formed specs). If FPs found, tighten patterns before ship.

The T11 anchor in the test suite ties acceptance criteria directly to calibration results — the suite fails if the calibration corpus produces FPs.

Prevention

Any future spec introducing a heuristic-based detection mechanism should:

1. Add a calibration task as a hard gate (not advisory) in the plan
2. Write a test fixture for the calibration check (T-last in the suite)
3. Document the corpus used and the acceptable FP threshold in decisions.md

Why Hard, Not Advisory

An advisory calibration produces a number but doesn't block. Developers under time pressure skip advisory steps. Making it a gate ensures the number is seen and acted on — and the test suite keeps it honest across future changes.

Calibration Outcome: Length-Gate the Strictest Sub-Rule (SPEC-162)

The gate tells you the FP rate; it does not tell you how to fix an over-firing heuristic. SPEC-162's FR-2a description-quality screen is the worked example.

The first heuristic required a positive signal — an explicit trigger/when-clause token — in every skill description. Against the real 127-skill corpus that hit 55% false positives: well-authored, specific descriptions (vt-d-activate, vt-d-2-plan, vt-d-spec-from-requirements) were flagged simply for not spelling out a "when" word they didn't need.

The wrong fix is to loosen the rule uniformly (that reintroduces false negatives). The right fix is to length-gate the strictest sub-rule — demand the positive signal only where its absence is genuinely ambiguous, and let unambiguous items pass without it:

• Fail on the unconditionally-bad shapes (empty, first-person, vague stem like "helps with X") regardless of length.
• Require the explicit when-clause only when the description is short (< 80 chars) — a terse description with no trigger is ambiguous; a long, specific one carries its own signal and passes without the token.

Result on the SPEC-162 corpus: 0 FP on the stratified known-good set, and the library flag count dropped from 70 to 15/127 — the 15 being genuine terse/vague descriptions, not specific-but-tokenless ones.

Generalizable rule: when a heuristic requires a positive signal in every item and the calibration corpus shows it punishing specific-but-terse content, gate the strictest sub-rule on the axis that distinguishes "ambiguous" from "self-evident" (here, length) rather than dropping the sub-rule. Stratify the calibration corpus across the real population's sub-shapes (SPEC-162: English-workflow / non-English / persona-domain) so the FP budget measures the distribution the heuristic will actually see, not a convenient monoculture. Document the residual FP posture (long specific descriptions pass without an explicit trigger) as an accepted trade-off, not a bug.