Skip to content

Judge served-model evidence per subagent run, never per session folder

Problem

SPEC-215 tiered every agent (model: opus|sonnet|haiku) and had to prove, from session transcripts, that each reviewer was served its declared tier after the deploy (T8, SC-7). The task picked its source session as "the oldest dir with mtime after the precondition check", and served_models.py read a whole session folder. Both assumptions broke on real data:

  • A folder's mtime says nothing about when its runs happened. One candidate folder showed 2026-10-08 21:19 only because the session had been moved. Its runs were from 2026-10-01, before the merge. Another folder held 14 pre-merge security-sentinel runs and would have been picked as soon as anything touched it.
  • Long-lived sessions mix pre- and post-deploy runs. Both real review sessions spanned several days. Over the whole folder the script exited 1, and every failing row was a run from before the deploy, served the old tier. Read on the review's own runs, it exited 0.
  • A run's dispatch can predate the precondition while its siblings follow it. In the chosen session the spec-compliance gate ran four hours before the precondition check and the seven parallel reviewers ran after it.
  • The alias follows the session's model, not a fixed ID. opus served claude-opus-5-5 in one session and claude-opus-5 in another, because each matched its parent session's model.

Pattern

  1. Select by run time. Take each subagent's first transcript timestamp (or its dispatch time), never the folder's mtime. A folder is a container, not an event.
  2. Read the review's own runs. Build a scratch copy of only that review's agent-<id>.meta.json / .jsonl pairs and run the checker on it. Record the manifest digest so the copy can be reproduced. Also record the whole-folder result and say which runs made it fail.
  3. Prove coverage separately. Re-run the selector (persona-select.sh) with the review's own --types and file list and compare it with the rows. A missing reviewer or a general-purpose stand-in fails, whatever the served models say.
  4. Compare families, record IDs. Judge an alias by its family (claude-opus-*) and record the exact served ID for comparison. A fixed-ID expectation fails whenever the session's model differs.
  5. A window is proven only by an input above it. "Served by an opus model" says nothing about a 1M window. Only a run without an override whose input exceeds 200,000 proves it. Without a read-to-the-end control, that run proves the window but not that a large input was read in full.

Prevention

Write selection rules over run timestamps from the start, and state the cut-off as a time, not as "the next session". Write any deviation into the record together with the rule it replaces. Re-measure before quoting, because new sessions keep arriving: SPEC-215's direct >200K security-sentinel run appeared a minute after the owner had ruled that one was not needed (specs/215-model-tiering-and-review-runtime-reduction/study/served-models.md, Decision 13).