# Sage vs. Claude: decision-latency benchmark

**Date:** 2026-07-23
**Run by:** Levanto Labs, manual comparison (not part of the automated fixture recorder)
**Models:** `levanto-sage-v0.5` · `claude-opus-4-8` · `claude-haiku-4-5` (verified as Anthropic's current flagship and fast-tier IDs on the run date)

This is the raw run behind the "Same question. Three models. Real time." section on the homepage. Every number below came from a live API call made on this date; nothing here is estimated.

## The question

A non-trivial policy call, not a coin-flip spam check:

> Given the policy and account facts above, may support auto-fulfill this request by emailing the 90-day conversation export today to the addresses named in the ticket, without waiting for Organization Owner approval or a formal DSAR?

## The content sent to every model

```text
Customer success ticket #CS-44182 (opened 2026-07-21 14:22 UTC)

Account: Acme Robotics GmbH (Enterprise, annual, €48,000 ARR, customer since 2023-11)
Requester: Lena Vogt, Head of Ops (verified domain @acmerobotics.de)
Channel: in-app chat → escalated to email

Message:
"We need to export our full conversation history for an internal investigation after a suspected account compromise last week. Please send a CSV of every message, tool call, and attachment filename for all workspaces under our org for the last 90 days to lena.vogt@acmerobotics.de and also CC security@acmerobotics.de. We need this today. If Self-Serve export is unavailable, just email the file."

Relevant policy excerpts from our Trust & Safety runbook (v4.2):
1. Data exports containing message bodies or attachment metadata for Enterprise customers require either (a) an organization Owner approving in-console, or (b) a verified legal hold / DSAR ticket with Legal tick-box.
2. Chat/email requesters who are not Owners may request exports, but fulfillment must wait for Owner approval unless the request is a formal DSAR filed through the Privacy portal.
3. Same-day email delivery of raw conversation exports is never permitted; exports are delivered only via the in-console Secure Export download (7-day link), never as email attachments.
4. Suspected account compromise: freeze new API keys and force SSO re-auth; do NOT broaden data egress until Security confirms the requester's identity via a video call or IdP step-up.
5. CC'ing an additional security@ alias does not substitute for Owner approval or Legal DSAR intake.

Known account facts:
- Lena Vogt is Billing Admin + Success contact, NOT Organization Owner.
- Organization Owners on file: kai.mueller@acmerobotics.de, sofia.nguyen@acmerobotics.de
- No open DSAR / legal-hold ticket for this org.
- Security has NOT yet completed identity step-up for this requester.
- Self-Serve Secure Export is available to Owners only.
```

## What Sage received

Sage takes content and a typed question directly, no prompt engineering:

```json
{
  "content": "<the content block above>",
  "question": {
    "id": "auto_fulfill_export",
    "kind": "yesno",
    "instructions": "Given the policy and account facts above, may support auto-fulfill this request by emailing the 90-day conversation export today to the addresses named in the ticket, without waiting for Organization Owner approval or a formal DSAR?"
  }
}
```

## What Claude received (full prompt, verbatim)

Both Claude models got this exact prompt over the streaming Messages API. It includes a calibration block asking the model to reason about its own confidence honestly, since chat LLMs are known to default to near-certainty:

```text
Decide yes or no. Reply with ONLY this JSON object and nothing else:
{"answer":"yes"|"no","confidence":<number between 0 and 1>}

No markdown. No prose. No other keys.

Confidence calibration guide (be honest, do not default to near-certainty):
- 0.95-1.00: the policy text explicitly and unambiguously decides this with no missing facts.
- 0.80-0.94: the policy points clearly to one answer, but requires combining 2+ rules or a small inference.
- 0.60-0.79: the policy is relevant but leaves some judgment call, edge case, or missing fact.
- 0.00-0.59: the policy is ambiguous, contradictory, or key facts needed to decide are missing.
Calibration means: if you say 0.9, you should be right about 90% of the time across similar cases. Do not inflate confidence just because you were able to produce an answer.

CONTENT:
<the content block above>

QUESTION:
Given the policy and account facts above, may support auto-fulfill this request by emailing the 90-day conversation export today to the addresses named in the ticket, without waiting for Organization Owner approval or a formal DSAR?
```

## How timing was measured

- **Sage:** `meta.latency_ms` returned by the `/decide` response body. This is the model's own report of its decision time and **excludes** network round-trip. We also recorded client wall-clock for the same calls, so you can see both.
- **Claude (Opus 4.8, Haiku 4.5):** the Anthropic Messages API does not return an internal generation-time field anywhere in the response body or headers. The only number available is client wall-clock over the streamed response, measured from request start to the final token, from a machine in Madrid. **This number includes network round-trip**, TLS handshake, and queueing, because there is no way to separate them from what Anthropic exposes.

This is the reason Sage is shown two ways and Claude is shown one way, not an attempt to flatter Sage: it's a direct consequence of what each API actually reports.

## Results (5 runs each, after 1 warm-up call per model)

### Sage — `levanto-sage-v0.5`

| Run | API latency (`meta.latency_ms`, no network) | Client wall-clock (incl. network) | Answer | Confidence |
| --- | --- | --- | --- | --- |
| 1 | 68.2ms | 247.5ms | no | 0.893 |
| 2 | 66.2ms | 230.3ms | no | 0.893 |
| 3 | 64.5ms | 227.2ms | no | 0.893 |
| 4 | 64.8ms | 256.9ms | no | 0.893 |
| 5 | 65.7ms | 238.2ms | no | 0.893 |
| **Median** | **65.7ms** | **238.2ms** | no | 0.893 |

### Claude Opus 4.8 — `claude-opus-4-8`

| Run | Time to first token | Wall-clock to usable JSON (incl. network) | Answer | Confidence |
| --- | --- | --- | --- | --- |
| 1 | 2375.1ms | 2532.0ms | no | 0.99 |
| 2 | 1859.1ms | 1990.6ms | no | 0.99 |
| 3 | 3967.3ms | 4337.4ms | no | 0.99 |
| 4 | 1115.3ms | 1393.1ms | no | 0.99 |
| 5 | 1365.4ms | 1613.1ms | no | 0.99 |
| **Median** | **1859.1ms** | **1990.6ms** | no | 0.99 |

### Claude Haiku 4.5 — `claude-haiku-4-5-20251001`

| Run | Time to first token | Wall-clock to usable JSON (incl. network) | Answer | Confidence |
| --- | --- | --- | --- | --- |
| 1 | 1732.8ms | 1943.6ms | no | 0.98 |
| 2 | 713.4ms | 820.9ms | no | 0.98 |
| 3 | 693.0ms | 1175.0ms | no | 0.98 |
| 4 | 593.3ms | 728.0ms | no | 0.98 |
| 5 | 1993.9ms | 2204.1ms | no | 0.98 |
| **Median** | **713.4ms** | **1175.0ms** | no | 0.98 |

## Summary

| Model | Latency (median) | What the number includes | Answer | Confidence |
| --- | --- | --- | --- | --- |
| Sage | **65.7ms** | model only, no network | no | **0.893** |
| Sage (reference) | 238.2ms | model + network | no | 0.893 |
| Claude Opus 4.8 | 1990.6ms | model + network (only number available) | no | 0.99 |
| Claude Haiku 4.5 | 1175.0ms | model + network (only number available) | no | 0.98 |

## Notes

- All three models agreed on the answer (**no**) across every run: the requester isn't an Owner, there's no DSAR, and email delivery of raw exports is against policy regardless.
- The calibration instructions did **not** move Claude's confidence at all: Opus stayed at 0.99 and Haiku at 0.98 across all 5 runs, identical to a prior run without the calibration block. Sage's confidence (0.893) reflects an actual trained calibration signal, not a self-reported guess: it does not default to near-certainty.
- Anthropic occasionally wraps Haiku's JSON output in a ` ```json ` fence despite an explicit "no markdown" instruction; Opus did not.