Skip to content

vt-c-skill-audit

Classify every skill in the core-standards manifest as evaluable, not-evaluable, or auto-keep, and write docs/skill-audit-inventory.{md,json}. Use before spending tokens on any benchmark sweep, or after adding/removing skills.

Plugin: core-standards
Category: Other
Command: /vt-c-skill-audit

Skill Audit — Evaluability Inventory

Reports the real shape of the core-standards skill library: which skills could be benchmarked, which have no eval suite yet, and which are kept regardless of any measurement.

Wave 1 measures evaluability only. It makes no keep-or-drop call about any skill, and it deletes, demotes, and unlinks nothing.

When to Use

  • Before a benchmark sweep — to know how many skills are actually measurable.
  • After adding or removing skills, when the pre-commit reminder flags the inventory as stale.
  • When you need the current eval-coverage number for planning.

Prerequisites

  • You are in the toolkit repo (or TOOLKIT_ROOT is set).
  • Nothing else. The inventory runs with no benchmark data present — that independence is deliberate.

Execution

Run from the toolkit repo root:

bash scripts/generate-skill-audit.sh

Useful flags:

Flag Effect
--skill-dir <path> Sweep the skills under <path> instead of the full manifest
--skill-file <path> Sweep one skill, given its SKILL.md
--tsv Print name<TAB>bucket<TAB>reason to stdout, write no files
--out-dir <path> Write the artifacts somewhere other than docs/
--count-only Print the parsed manifest entry count and exit

Then report to the developer, from the script's output and the artifact header only:

  1. The artifact paths.
  2. The headline figures: the four bucket counts and all four shares — including the evaluable share, which is the number this spec exists to establish. The two auto-keep fractions must stay separate figures (D8), never merged into one auto-keep number.

Do not read skill bodies into context to summarise them — the sweep already happened in the script, and re-reading every SKILL.md in the manifest buys nothing but tokens.

Buckets

Bucket Meaning
evaluable The skill carries an eval suite and could be benchmarked
auto-keep-private-data The skill supplies context the base model cannot have
auto-keep-toolkit-internal The skill wires into the toolkit's own registries
not-evaluable No eval suite exists yet — flagged for suite creation

Precedence runs top to bottom; every skill lands in exactly one bucket.

The predicate that assigns the two auto-keep buckets is a marker table inside plugins/core-standards/bin/generate-skill-audit.sh. It is not restated here — a predicate written in prose silently does not run, and two copies drift apart. Changing the table is a code change with fixtures, not a wording change.

Edge Cases

  • A high auto-keep share — that is a finding to report, not a failed audit.
  • A low evaluable share — expected. It measures eval coverage, nothing else.
  • Scoped sweep with --skill-dir — skills outside the manifest get the category (scoped).

Error Handling

The script is advisory and always exits 0. If sources are missing, the affected skills simply do not appear in the sweep — check the reported skill count against the manifest before trusting a run.

Integration Points

  • Reads plugins/core-standards/.claude-plugin/skill-symlinks.manifest for names and categories.
  • Probes tests/evals/<skill>/ for eval suites (SPEC-098's surface). Benchmark output is never read.
  • Emits docs/skill-audit-inventory.{md,json} — a sibling of the SPEC-125 orchestration map, with its own schema version.
  • Pre-commit Check E reminds you to regenerate when the manifest or a SKILL.md changes.

Success Criteria

  • Both artifacts exist and the reported skill count matches the manifest.
  • Every skill appears in exactly one bucket.
  • Both auto-keep fractions are reported as separate figures.