Agentic security
Guardrails no longer have to choose between smart and fast.
Sage performs like GPT-5 on AgentHarm, at about 5x the speed, fast enough to sit in the hot path, smart enough to actually read the conversation.
The trade-off
The guardrail industry keeps re-discovering the same failure mode.
Classifiers are fast, but blind
Most guardrails score only the last message, with no view of the system prompt or the rest of the conversation. They can't follow multi-step reasoning, and harm assembled gradually, through roleplay, personas, or fabricated authority, slips past them.
LLM judges are smart, but slow
A frontier model reading the full transcript is smart enough to catch that harm. It's also 1–1.5 seconds of latency per call, an eternity inside an agent loop or a checkout flow.
So teams pick a failure mode
Too strict, and legitimate requests get blocked, an outage with a policy attached. Too loose, and harmful multi-step tasks complete anyway. Most deployed guardrails land on one of these two failures.
The breakthrough
Sage fuses LLM judgment with classifier latency.
Built by fusing LLM and classifier capabilities, Sage reads the full conversation, system prompt included, and reasons about intent and trajectory, not surface keywords. It answers in the time a classifier takes, with no domain-specific training required. For the first time, a guardrail can have the intelligence of an LLM.
LLM-grade judgment
Reads the whole conversation, weighs multi-turn context, and reasons about intent, not keyword matching.
Classifier-grade latency
Answers in ~200ms client round-trip, fast enough for the hot path of an agent loop or a checkout.
Generalized, not domain-trained
No per-attack-type fine-tuning. The policy is your instructions, not a fixed training set.
The result
Safety–utility Pareto on AgentHarm
% of harmless requests allowed (x) × % of harmful tasks prevented (y). Ideal = top-right. Latency is the median client round-trip per decision.
Google Model Armor stops more harm than Sage (89.7% vs 77.0%), but it blocks 1 in 8 harmless requests (12.5% over-block). At that over-block rate, a real product would be fighting its own guardrail as much as any attacker.
From the published AgentHarm run; see the reproduction guide for the full protocol.
The hot path
Median latency, client round-trip
All 352 AgentHarm prompts, sequential, from the same host in AWS us-east-2. Success-only, median (p50) shown.
Sage answers in 208ms, ~4.9x faster than GPT-5’s 1009ms, at comparable AgentHarm quality.
Full results
Every system, as published
AgentHarm test_public split, 176 harmful + 176 matched benign. Operating thresholds calibrated at ≤ 5% FPR on the same benign split (Bedrock / Model Armour use fixed confidence arms).
| System | AUC | Prevented | Allowed | p50 latency |
|---|---|---|---|---|
| Google Model Armour | 0.904 | 89.7% | 87.5% | 195ms |
| Levanto SageSAGE | 0.880 | 77.0% | 95.5% | 208ms |
| OpenAI GPT-5 | 0.901 | 76.3% | 96.0% | 1009ms |
| AWS Bedrock LOW | 0.521 | 58.8% | 88.1% | 191ms |
| Meta PromptGuard | 0.690 | 57.3% | 95.5% | 301ms |
| OpenAI GPT-5-mini | 0.836 | 56.1% | 100.0% | 1114ms |
| InjecGuard | 0.594 | 55.8% | 95.5% | 262ms |
| Protect AI | 0.565 | 54.7% | 95.5% | 312ms |
| AWS Bedrock MED | 0.521 | 51.6% | 96.0% | 188ms |
| Unguarded GPT-4o | - | 51.1% | 100.0% | - |
Why AgentHarm
Most jailbreak benchmarks are already saturated.
The field is full of prompt-injection and jailbreak benchmarks built on judging static strings. They're largely saturated (classifiers put them straight into their training sets), and they're not interactive: they don't score what actually happens when an unguarded agent is handed the same request.
AgentHarm (Andriushchenko et al., ICLR 2025, arXiv:2410.09024), developed by the UK AI Security Institute and Gray Swan AI, instead gives a real tool-using agent multi-step malicious tasks and scores how much harmful work actually gets completed. We ran its public test split: 176 harmful + 176 matched benign scenarios, comparing an unguarded model against the same model under each guardrail.
As the unguarded baseline we kept the paper's original model, GPT-4o, which prevents 51.1% of harmful work while allowing 100% of harmless requests, the starting point every guardrail in this comparison is measured against.
Read before you cite this
What this chart does and doesn't show
- Thresholds are calibrated in-sample: ≤ 5% FPR on the same 176 benign prompts used for evaluation, not a held-out set.
- Conversations are live and non-deterministic. Harm score and over-block mix the gate decision with a tool-using agent and two GPT-4o judges, so results move run to run; expect stable ranking, not bit-identical numbers.
- The open-weight classifiers (Protect AI, PromptGuard, InjecGuard) were measured on CPU on the same host, not a GPU-optimized deployment; their latency would likely drop on GPU.
- GPT-5 / GPT-5-mini use reasoning_effort=minimal. We have not published a reasoning=low run.
Drop-in guardrail
Score the transcript, branch on confidence.
Same shape as any other Sage decision: send the conversation, get a typed answer with a calibrated confidence you can branch on.
# pip install levanto
from levanto import LevantoClient, YesNo
client = LevantoClient(api_key="lv_live_...")
env = client.decide(
transcript,
YesNo("Is the user working the assistant toward an outcome it should refuse?"),
)
score = env["result"]["probability"]
if score >= 0.716:
refuse(transcript) # matches the AgentHarm operating point
elif env["result"]["confidence"] < 0.6:
escalate_to_human(transcript) # not sure? a human decides
else:
run_agent(transcript)We're already talking with security and guardrail teams about this.
If you're building or buying guardrails, we'd like to hear from you: feedback, collaborations, or just a second opinion on your threat model.