Three-Pass Spec Validation¶

Problem¶

Specs that look sound under merit validation can still be fundamentally wrong. A spec can pass every box of the activation-time merit check — concrete evidence cited, named consumers, measurable success metric, no constitution conflict — and still be a bad spec to build. The reasons are not the ones merit validation looks for.

Observed (2026-05-11, SPEC-121): A meta-strategy spec called "Adoption Strategy — beyond rolf-only-user" was activated. Merit validation flagged it SOUND-with-concerns. The concerns were tactical (declare deps, scope shape). On surface it was fine: it cited a direct verbatim quote from the source 1:1, it named the personas, it had a 6-month success metric.

The actual problems, all missed by merit validation:

The spec conflated two distinct topics — toolkit-adoption AND a team planning surface — bound together only by a single rhetorical sentence in the source intake ("Niclas's feature wishes are what he'd need to adopt"). Two perfectly valid topics in one spec is incoherent: the success metric of one tests the build of the other.
The success metric tested habit; the build tested activation. Nothing in US-1–US-4 created a return-trigger. The Spec was internally inconsistent and merit validation didn't notice because it checks each piece independently.
A subsequent strategic-mentor agent hallucinated context. When dispatched for strategic review, the AI agent produced a "10× alternative" referencing Asana and Slack — neither of which the company uses. The agent had no domain ground truth and didn't know it didn't know.

Outcome: spec withdrawn the same day, split into two new intakes for fresh qualification. No code was written. Cost: ~30 minutes. Value: avoided weeks of building wrong things.

Symptoms when this is violated:

Spec passes activation, gets into planning, and the plan phase repeatedly stops to ask strategic questions that should have been answered earlier.
Build phase produces deliverables that no acceptance test really validates — because the success metric was measuring something the build doesn't deliver.
Post-mortems reveal "we built the wrong thing" or "we should have split this into two specs."
A reviewer / mentor agent generates plausible-sounding analysis that turns out to assume tools, processes, or organizational facts that don't exist.

Pattern: Three independent validation passes¶

Each pass catches a different class of fault. Skipping any one creates a specific blind spot. Run them in order; later passes are cheap only because earlier passes have done their job.

Pass 1 — Merit validation (at activation)¶

Where: /vt-c-activate Step 7. Mechanical, fast (<1 min, no agent dispatch).

What it catches: - Constitution conflicts (does this violate stated principles?) - Missing evidence (is the problem real, with citations?) - Unclear value (do the deliverables have named consumers and a measurable goal?)

What it cannot catch: - Two valid topics fused into one spec. - A spec where each piece looks fine but the combination is incoherent. - Hidden assumptions about motivation, capability, or adoption that the spec treats as settled.

This is a sanity gate. It catches drift, sloppy specs, missing references. It is not adversarial.

Pass 2 — Strategic adversarial review (at shape)¶

Where: /vt-c-shape Step 2.5. Dispatches an agent (e.g. product-challenge-reviewer or strategic-mentor) to deliberately attack the spec.

What it catches: - The hidden bet (the assumption the spec is staking everything on, which if wrong renders the work useless). - Failure modes ranked by likelihood × cost. - "You're building the wrong thing" framing — the more ambitious alternative, the cheaper alternative that tests the bet first. - Internal inconsistencies (e.g. success metric measures habit, build produces only activation).

What it cannot catch: - Domain facts the agent doesn't know. The agent has only the spec, the constitution, and the intake source — it does NOT know which tools the company uses, who actually adopts what, or the broader org context. Strategic review is sharp on logic and naive about reality.

This is the adversarial pass. It is what most often surfaces "this spec is fundamentally wrong" findings.

Pass 3 — Ground-truth check (by the human)¶

Where: After Pass 2, before locking shape decisions. No skill, no agent — the human reads the agent's output and corrects any claims that assume facts the agent couldn't know.

What it catches: - Agent-hallucinated tools, processes, names, org structures. - Recommendations grounded in wrong assumptions about the team's actual workflow. - Conflations the spec author themselves made (the human knows their own context).

What it requires: - The human actually reads the agent's strategic review with skepticism rather than rubber-stamping. The agent will produce plausible analysis even when its premises are wrong; only a domain expert can tell.

This is the cheapest pass and the one most often skipped. It is also the one that turned SPEC-121 from "interesting strategic concerns" into "this is two specs and one of them isn't toolkit-shaped."

How the three passes interact¶

                  ┌────────────────────────────┐
  /vt-c-activate  │ Pass 1 — Merit             │   catches: drift,
   Step 7         │  • constitution            │            sloppy specs,
                  │  • evidence                │            missing refs
                  │  • value                   │
                  └─────────────┬──────────────┘
                                ↓ SOUND or SOUND-with-concerns
                  ┌─────────────┴──────────────┐
  /vt-c-shape     │ Pass 2 — Strategic review  │   catches: hidden bet,
   Step 2.5       │  • hidden bet              │            failure modes,
                  │  • failure modes           │            wrong shape,
                  │  • 10× and <2-week alts    │            internal incoherence
                  └─────────────┬──────────────┘
                                ↓ verdict + analysis
                  ┌─────────────┴──────────────┐
  human review    │ Pass 3 — Ground truth      │   catches: agent
   (no skill)     │  • is the analysis based   │            hallucination,
                  │    on reality?             │            author's own
                  │  • what's the agent        │            conflations
                  │    assuming?               │
                  └─────────────┬──────────────┘
                                ↓ shape decisions locked OR spec withdrawn
                            /vt-c-2-plan

The pattern is layered defense: cheap mechanical pass first, expensive adversarial pass second, free expert pass third. Each pass has different blind spots; the union covers most spec faults.

Specific spec antipatterns this pattern catches¶

Two-topic conflation. A spec that bundles two valid concerns. Signal: Open Decisions section is long and the items don't share a coherent thread. Caught by: Pass 2 ("this looks like two specs") or Pass 3 ("oh, those are different problems").
Habit-vs-activation mismatch. Success metric tests sustained behavior over time, but US-1–US-N only deliver one-time setup. Caught by: Pass 2 (internal inconsistency).
Hidden motivation assumption. Spec assumes "if we build it, they will use it" — treats a "won't" as a "can't." Caught by: Pass 2 (hidden bet) plus Pass 3 (the human knows whether the assumption is real).
AI-hallucinated context in reviews. Review agent recommends approaches based on tools/processes that don't exist at the company. Caught only by: Pass 3 (the human notices).

When to use this pattern¶

Every non-trivial spec. The cost of all three passes combined is small; the cost of building the wrong spec is large.
Especially for meta-specs, strategy specs, adoption specs, and any spec whose deliverable_types includes research or project-plan. These are the most likely to have hidden bets and conflated topics.
Especially when the spec source is a transcript or 1:1 conversation. Rhetorical glue in human conversation easily fuses distinct ideas; specs derived from it inherit the fusion unless someone attacks it.

When NOT to use this pattern (or to use a lighter version)¶

Pure technical specs with concrete code deliverables and a single user story. SPEC-124 (defensive toolkit install) was sound under merit alone — no strategic mystery, no motivation assumption, no conflation risk. Pass 2 and Pass 3 would have been theater.
Bugfixes invoked via /vt-c-activate BUG-NNN. Bugs do not need strategic review; they need a reproducer.

A simple heuristic: if the spec contains an "Open Decisions" section with 3+ items, OR if deliverable_types is anything other than [code], OR if the problem statement quotes a human conversation rather than citing a code symptom — run all three passes. Otherwise Pass 1 alone is enough.

Status — automated 2026-05-21 (SPEC-137): This textual heuristic is now computed at activation time by plugins/core-standards/skills/activate/scripts/shape-worthiness-score.py. The script implements the rule on five locked criteria (the three above plus depends_on:[] with ≥2 hidden cross-spec refs and ### US- count ≥3) and emits a 0–5 score with contributing factors. /vt-c-activate Step 7 substep 4.5 invokes the script and surfaces /vt-c-shape as the recommended next action when the score crosses 2. This pattern doc still describes the intent — the script enforces it deterministically across activations.

Cost / benefit¶

SPEC-121 case (2026-05-11): - Pass 1 (merit): ~2 min. Verdict: SOUND-with-concerns. - Pass 2 (strategic-mentor): ~30 sec wall time, ~57k tokens. Verdict: NEEDS REVISION. - Pass 3 (ground truth): ~2 min reading. Verdict: contains hallucinated tools, also reveals conflation. - Outcome: spec withdrawn, two clean intakes filed, ~30 min total.

If the spec had reached /vt-c-3-build instead: estimated 2–5 days of work on a spec that conflated two topics and had no return-trigger built. Cost-of-skipped-passes would have been on the order of 100×.

Reference¶

Spec: SPEC-121 (specs/121-adoption-gap-rolf-only-user/) — rejected 2026-05-11; first documented case of this pattern.
Replacement intakes: intake/inbox/2026-05-11-toolkit-entry-points-for-non-power-users.md, intake/inbox/2026-05-11-planning-surface-jira-vs-markdown-vs-custom.md.
Skills involved: /vt-c-activate (Pass 1), /vt-c-shape (Pass 2), human review (Pass 3).
Open question: should Pass 2 (/vt-c-shape) be merged into Pass 1 (/vt-c-activate)? See docs/decisions/ for the decision when made.