Skip to content

An external verdict stands in for a local gate only when "green" is read from the protected base

Problem

SPEC-221 let /vt-d-4-review take its test gate from GitHub Actions instead of a local full run. A CI conclusion of success looks like the same evidence as a local PASS, but it is only as good as the definition of the job that produced it — and that definition lives in the branch being reviewed. A branch that appends || true to a step in guard-suite.yml, stubs run-tests.sh, or lowers guard-count.txt gets a green run for code that never passed. A CI gate records no test counts, so nothing in the gate file shows it (review passes 2–5 executed each of these).

Pattern

  1. Read the definition from the protected base, never from the branch. The evaluator reads .github/test-gate-ci.txt from refs/remotes/origin/main, resolved with show-ref --verify (a tag named origin/main shadows a bare rev-parse origin/main, pass 2).
  2. Require every file that decides "green" to equal the base. The definition, each listed workflow and each unchanged path (runners, the libraries they source, guard baseline and count) are diffed against the base; any difference is "not applicable", not "pass". A branch that needs to change them takes the local path.
  3. Derive the guarded set where you can, and pin it in a test. T44 derives the libraries both runners source and fails when one is not unchanged. That derivation caught the next instance on its own: merging main brought lib/negative-control.sh into run-all.sh, and T44 went red for exactly that file.
  4. Match the run to the exact commit. Only headSha == full HEAD SHA, only events that ran the branch's own HEAD (workflow_dispatch, push), the newest run per workflow decides, and only completed/success passes. Every other shape is "incomplete", never a pass.
  5. Remove the old external PASS before checking anything. A run killed by a timeout or signal must not leave the previous PASS standing (pass 6): with --write-gate, an earlier source: ci gate is deleted first, and a new gate is written only on a definite pass or fail. Bound the whole evaluation below the caller's timeout so it can always print its verdict.
  6. Fail closed on everything you cannot classify. Unknown conclusion, non-string event, bad run id, unreadable output, internal error: exit 2 ("unavailable") with a reason, and never fall back to a local run silently — that would recreate the load the external source exists to avoid.

Prevention

  • When a gate starts trusting a result it did not produce, list the files that define that result's meaning and ask who can edit them. If the answer includes the branch under review, pin them to the protected base.
  • Treat "which files does the runner load" as data a test derives, not a list a person maintains; the list grows on every merge.
  • Probe the boundary values of the match (one character off the SHA, a 7-character prefix, run id -1 and 0), and mutation-test each fail-closed branch: in pass 7 the budget-exhausted branch and the per-call timeouts still survived mutation.