Agentic security
Guardrails no longer have to choose between smart and fast.
Sage reads the whole conversation and decides in the time a classifier takes.
The trade-off
Fast and blind, or smart and slow.
Classifiers are fast, but blind
Most score only the last message, so harm built up through roleplay, personas or fake authority slips past.
LLM judges are smart, but slow
A frontier model catches that harm, at 1–1.5 seconds per call.
So teams pick a failure mode
Too strict blocks real users. Too loose lets harmful tasks finish.
LLM judgment at classifier speed.
LLM-grade judgment
Reads the whole conversation and judges intent, not keywords.
Classifier-grade latency
Fast enough for an agent loop or a checkout.
Generalized, not domain-trained
Your instructions are the policy. No per-attack training.
The result
Safety–utility Pareto on AgentHarm
Harmless allowed (x) vs harm prevented (y). Top-right is best.
Google Model Armor stops more harm (89.7% vs 77.0%) but blocks 1 in 8 harmless requests. At that rate, a product fights its own guardrail.
The hot path
Median latency
All 352 prompts, one at a time, from AWS us-east-2. Successful calls only.
Full results
Every system, as published
AgentHarm public test split: 176 harmful, 176 matched harmless.
| System | AUC | Prevented | Allowed | p50 latency |
|---|---|---|---|---|
| Google Model Armor | 0.904 | 89.7% | 87.5% | 195ms |
| Levanto SageSAGE | 0.880 | 77.0% | 95.5% | 208ms |
| OpenAI GPT-5 | 0.901 | 76.3% | 96.0% | 1009ms |
| AWS Bedrock LOW | 0.521 | 58.8% | 88.1% | 191ms |
| Meta PromptGuard | 0.690 | 57.3% | 95.5% | 301ms |
| OpenAI GPT-5-mini | 0.836 | 56.1% | 100.0% | 1114ms |
| InjecGuard | 0.594 | 55.8% | 95.5% | 262ms |
| Protect AI | 0.565 | 54.7% | 95.5% | 312ms |
| AWS Bedrock MED | 0.521 | 51.6% | 96.0% | 188ms |
| Unguarded GPT-4o | - | 51.1% | 100.0% | - |
Why AgentHarm
Most jailbreak benchmarks are already saturated.
Most jailbreak benchmarks judge static strings, sit in classifier training sets, and never score what an agent actually does.
AgentHarm (Andriushchenko et al., ICLR 2025, arXiv:2410.09024), developed by the UK AI Security Institute and Gray Swan AI, instead gives a real tool-using agent multi-step malicious tasks and scores how much harmful work actually gets completed.
The unguarded baseline is the paper's original model, GPT-4o.
Read before you cite this
What this chart does and doesn't show
- Thresholds are calibrated in-sample: ≤ 5% FPR on the same 176 benign prompts used for evaluation, not a held-out set (Bedrock and Model Armor use fixed confidence settings).
- Conversations are live and non-deterministic. Harm score and over-block mix the gate decision with a tool-using agent and two GPT-4o judges, so results move run to run; expect stable ranking, not bit-identical numbers.
- The open-weight classifiers (Protect AI, PromptGuard, InjecGuard) were measured on CPU on the same host, not a GPU-optimized deployment; their latency would likely drop on GPU.
- GPT-5 and GPT-5-mini ran at reasoning_effort=minimal.
Drop-in guardrail
Score the transcript, branch on confidence.
import os, httpx
resp = httpx.post(
"https://sage.levanto.ai/decide",
headers={"Authorization": f"Bearer {os.environ['SAGE_API_KEY']}"},
json={
"content": transcript,
"question": {"id": "needs_refusal", "kind": "yesno",
"instructions": "Is the user working the assistant "
"toward an outcome it should refuse?"},
},
)
result = resp.json()["result"]
if result["probability"] >= 0.716:
refuse(transcript) # matches the AgentHarm operating point
elif result["answer"] is None:
escalate_to_human(transcript) # not sure? a human decides
else:
run_agent(transcript)