Heuristic Calibration as a Hard Delivery Gate

Problem

Pattern-matching heuristics in workflow gates can fire on well-formed inputs, eroding user trust in the gate over time. If the false-positive rate is not validated before shipping, the gate becomes noise — users learn to dismiss it.

Discovered when building gray-area-heuristic.py for SPEC-127: the calibration task was initially advisory ("run and note the results"), but the false-positive risk warranted elevating it to a ship-blocker.

Solution

Include a mandatory calibration task in plan.md that:

1. Runs the heuristic against a real corpus (5–10 existing spec/PRD files that are known-good — i.e., already shaped or clearly well-formed)
2. Records the false-positive count and rate
3. Requires that rate to be acceptable before the spec can be marked complete

The calibration task should be listed in tasks.md as type: gate — not type: task — so it cannot be skipped without an explicit gate bypass.

Implementation Example (SPEC-127)

# tasks.md excerpt
- id: T11
  title: Calibration gate — 0 FP on well-formed corpus
  type: gate
  acceptance: |
    Run gray-area-heuristic.py against specs/[N]-*/spec.md files with
    status=completed. Verify 0 false positives (no gray-area prompts
    for well-formed specs). If FPs found, tighten patterns before ship.

The T11 anchor in the test suite ties acceptance criteria directly to calibration results — the suite fails if the calibration corpus produces FPs.

Prevention

Any future spec introducing a heuristic-based detection mechanism should:

1. Add a calibration task as a hard gate (not advisory) in the plan
2. Write a test fixture for the calibration check (T-last in the suite)
3. Document the corpus used and the acceptable FP threshold in decisions.md

Why Hard, Not Advisory

An advisory calibration produces a number but doesn't block. Developers under time pressure skip advisory steps. Making it a gate ensures the number is seen and acted on — and the test suite keeps it honest across future changes.