Skip to content

Microsoft RAMPART + Clarity — Open-source Agent Safety Testing and Pre-Implementation Assumption Checks

Summary

Microsoft open-sourced two tools that together cover both ends of the agent-development workflow: RAMPART (github.com/microsoft/RAMPART) is a PyRIT-based, pytest-style framework that encodes red-teaming scenarios — especially cross-prompt-injection — as CI tests with explicit statistical pass policies; Clarity (github.com/microsoft/clarity-agent) is a structured design-assumption checker that runs before implementation begins, writing failure analyses and decision records to a versioned .clarity-protocol/ directory. RAMPART maps onto /vt-c-4-review + test-driven-development; Clarity maps onto /vt-c-shape + /vt-c-2-plan + spec-flow-analyzer.

Key Details

  • RAMPART — agent tests as CI assets. Tests describe a threat-model scenario, connect to agents via adapters, orchestrate interactions, and evaluate outcomes. Pass/fail signal plugs into CI like any other integration test.
  • Statistical pass policies. "This action must be safe in ≥80% of runs" — a different primitive from the deterministic assertions in our current TDD setup, designed for LLM probabilistic behavior.
  • Composable evaluators with boolean logic inspecting tool invocations, side effects, and whether actions remain in expected bounds. Closer in spirit to our agent-tool guardrails than to plain integration assertions.
  • Cross-prompt injection coverage out of the box — agents retrieving poisoned content from emails, tickets, or documents and acting on embedded instructions.

  • Red-team-to-test pipeline. Findings from engagements become permanent runnable tests; engineers own the failures.

  • Clarity — design-stage failure analysis. Runs as desktop app, web UI, or embedded in coding agents. Structured Q&A covering problem clarification, solution exploration, failure analysis, and decision tracking.

  • Multi-thinker failure analysis. Independent AI "thinkers" examine the design through security, human-factors, adversarial, and operational lenses. Reconciled into a management plan. Mechanically similar to our parallel reviewers in implementation-orchestrator, but applied at the spec stage.
  • .clarity-protocol/ directory committed and reviewed like source code. Direct parallel to our specs/{SPEC-NNN}/ convention.
  • Staleness tracking across dependent docs — relevant to our spec/state-drift work (SPEC-141).

Why Rolf Thinks This Matters

Further Reading