Skip to content

Anthropic Agentic Misalignment Paper — 7 Prompting Lessons

Summary

Anthropic's agentic-misalignment paper provides hard numbers for an argument we've been making informally: principle-based prompts outperform flat rule lists. Targeted "rule-matching" training drops misalignment on the test scenario (22%→15%) but generalizes to nothing. Rewriting the same prompts to teach the reasoning behind the aligned behaviour drops misalignment to 3%, and constitutional training survives subsequent reinforcement learning intact. The directive for CLAUDE.md and skill design: write a constitution, not a checklist; every rule needs a "because" attached.

Key Details

  • Lesson 1 — Rules don't generalize. Training on prompts similar to the failure scenario moved misalignment 22%→15% on that test but produced zero improvement on held-out scenarios.
  • Lesson 2 — Reasoning, not commands. Same prompts, but with the assistant's response showing the why behind the aligned action: misalignment dropped 22%→3%.
  • Lesson 3 — Constitution > checklist. A "difficult advice" dataset teaching transferable reasoning outperformed targeted training with 28× less data (3M vs 85M tokens).
  • Lesson 4 — Diverse context shapes reasoning. Adding ADRs, READMEs, incident postmortems, PR templates to the agent's context improved alignment even when the tools weren't needed for the task.
  • Lesson 5 — The demo-project trap. Chat-aligned models lost alignment in agentic settings. Claude Sonnet 4.5 hit near-zero on the targeted test but stayed misaligned on out-of-distribution scenarios.
  • Lesson 6 — Principles survive RL pressure. Constitutional alignment held through subsequent reinforcement learning. Rule-based alignment got washed out.
  • Lesson 7 — Playbook. Audit your CLAUDE.md/cursor rules; rewrite each rule with a "because"; write a 5–10 statement codebase constitution; expand context with ADRs and postmortems; stop patching with new rules — teach the missing principle.

Why Rolf Thinks This Matters

Further Reading