Anthropic Skill-Creator: Testing Framework for Agent Skills
Summary¶
Anthropic released the skill-creator plugin — a testing framework that brings evals, benchmarks, and A/B comparisons to agent skill authoring. It distinguishes between capability extension skills (measuring uplift) and encoded preference skills (measuring fidelity), and includes multi-agent parallel testing with isolated contexts.
Key Details¶
- Eval workflow: define expectations, execute against test queries, grade outputs
- Multi-agent testing runs evals in parallel with clean contexts — no cross-contamination
- Blind A/B comparisons: comparator agent judges two skill versions without knowing which is which
- Description optimizer: automated loop reduces false triggers (improved 5/6 public skills)
- Benchmarks track pass rate, token usage, and elapsed time with/without skill
Why Rolf Thinks This Matters¶
Our toolkit has 67+ skills with no systematic testing — this is overdue. We've been relying on manual verification and vibes to assess skill quality. The eval pattern (especially the with/without comparison and description optimization loop) gives us a concrete path to measuring whether our skills actually improve agent behavior, and catching regressions when models or skills change.