Self-Referential Artifact Tests

The failure

A generator emits a tracked artifact — docs/foo-inventory.json, a generated map, a rendered report. Its test suite needs that artifact to assert on, so the suite runs the generator and then reads the artifact from its default output path, which is the committed copy.

Two things go wrong at once, and the second hides the first:

1. The assertions no longer test the generator. The committed artifact is already correct — that is why it was committed. Every assertion passes on the file's existing contents, whether or not the generator ran.
2. The invocation swallows failure. Because a failing generator would abort the suite under set -e, the call gets wrapped in || true. Now a deleted, renamed, or crashing generator is indistinguishable from a working one.

In SPEC-156 this made four test groups (T3, T5, T6, T10 — the entire artifact surface) pass with the generator deleted. It survived a full review pass: Pass 1 found and removed 25 lines of genuinely vacuous assertions elsewhere in the same file while leaving this in place, because the vacuous lines looked tautological and this looked like ordinary setup.

The dirty working tree after every test run is the visible symptom. It gets rationalised as a known annoyance ("expect a timestamp delta") rather than read as what it is: the test and the fixture are the same file.

The pattern

Generate into a scratch directory, and assert the exit status.

TMP_OUT=$(mktemp -d)
trap 'rm -rf "$TMP_OUT"' EXIT
OUT_MD="$TMP_OUT/inventory.md"        # never $TOOLKIT_ROOT/docs/...

run_generator() {                      # never `|| true`
  local label="$1"; shift
  local rc=0
  bash "$GENERATOR" --out-dir "$TMP_OUT" "$@" >/dev/null 2>&1 || rc=$?
  assert_equals "$rc" "0" "$label"
}

Three properties follow, and each is worth asserting explicitly:

• The suite fails if the generator is gone. That is the whole point.
• The working tree stays clean. Verifiable directly, and a much better regression guard than a comment: BEFORE=$(git hash-object <artifact>); <run suite>; [ "$BEFORE" = "$(git hash-object <artifact>)" ]
• The generator needs an output-path flag. If it has none, add one. A generator that can only write to one hard-coded location is untestable, and the flag pays for itself immediately.

Prerequisite: a real output flag, and a caller for it

SPEC-156's --out-dir was itself flagged for removal during review as an unused flag with "zero callers or tests" — correct at the time, and exactly backwards. The flag was not dead; the test suite was failing to use it. Fixing the test gave the flag its caller and closed both findings at once.

Read a lone unused output-path flag on a generator as a missing test, not as dead code.

The near-miss that proves the shape

Writing a follow-up test for this same spec — one asserting that two generators agree on a count — the obvious implementation invoked the sibling generator to ask it for the number. That generator had no --count-only, so it did a full run and rewrote its own tracked artifact as a test side effect: the identical bug, reintroduced in the commit that fixed it, minutes later.

The replacement compares the duplicated source literal directly and reads the count from the committed artifact — both read-only.

If a test needs a number from another tool, prefer reading it from that tool's committed output or source over invoking the tool. Invocation is a write.

Related

• test-enforced-redaction-claims.md — the sibling failure: a guarantee with no fixture that seeds the thing being guaranteed against. Both reduce to does this assertion have a path to red?
• structural-test-assertions.md — pair every positive with a negation.
• dogfooding-meta-acceptance.md — when a command's own next run is the acceptance test, that is the safe version of self-reference: the output is observed, not overwritten.