The Case Against Skills
Summary¶
Every makes the counterintuitive case that as frontier models improve, elaborate skill libraries can make an agent worse — spending tokens to force the model down a prescribed path it would navigate better on its own. Skills earn their keep only when they supply something outside public training data: private context, custom tools, company templates, or specific workflows.
Key Details¶
- SWE-Skills-Bench: across 49 SWE skills, 39 had no impact, 3 reduced performance, and only 7 helped; the worst inflated token use by 451%. The 7 winners all supplied specialized private knowledge (e.g. financial-risk formulas).
- Mike Taylor: skills that patched Opus 4.8 weaknesses degraded Fable 5 — "you're fighting the weights of the model." Mirrors the GPT-4 era, when better instruction-following retired earlier workarounds.
- Skills have a shelf life: instructions compensating for a model blind spot go redundant (or harmful) once a newer model absorbs the capability.
- Skills audit ("try it this week"): Keep private-context/tool/template/workflow skills; Retest those compensating for current model weakness; Retire anything without demonstrable lift.
- Evaluation method: build a golden data set of ideal outputs, generate an LLM judge, and run with/without sanity checks. One durable public skill cited: OpenClaw's
autoreview(sends a diff to a second model before merge).