Model routing
A router as smart as a frontier model, fast enough to run on every request.
Model routers have to be super fast and cheap enough not to eat into the savings, which is why they're built from simple classifiers, not frontier models. Sage breaks that trade-off: closed-ended judgment with a calibrated confidence on every answer, fast enough to sit in front of every call.
The routing trade-off
Routers must be fast and cheap. That's why they can't be smart.
The router is on the hot path
It runs before every single call, adding latency to requests that haven't started yet. Anything that takes a second is a tax on your whole product, not a one-off cost.
It has to pay for itself
A router exists to save money. Put a frontier model in the routing seat and the routing bill eats the savings you were routing to capture. So teams reach for tiny encoders and keyword classifiers.
But small classifiers can't judge difficulty
Deciding which model can solve a prompt means reasoning about the prompt: how much capability it needs, how obscure the knowledge is, whether it needs real calculation. An embedding of the surface text doesn't know any of that.
Why Sage fits routing
Sage was built for exactly this shape of problem.
A router needs closed-ended judgment, fast, cheap, and with a usable notion of uncertainty. That is the entire design of Sage, which is why it can sit in the routing seat without the usual compromise between smart and fast.
Frontier judgment at classifier speed
Reasons about what a prompt actually requires, fast enough to sit in front of every request.
Calibrated confidence on every answer
The part that makes routing work: uncertainty becomes a feature your router can use, not a hidden failure.
Cheap enough to run on every request
One batched call per prompt, ~$0.0006, small enough to disappear against the savings.
Head-to-head vs OpenRouter
Levanto vs OpenRouter Auto Beta on AIME
60 AIME math problems, same prompts on both sides, unrestricted model pool. OpenRouter's Auto Beta is the direct comparison: a production router built from classifiers, running live against our Sage-based router.
Both routers ran at three settings: quality-first, balanced, and cost-first. Levanto costs include the Sage routing fee; OpenRouter costs are billed API usage with no separate routing fee.
Trade a little quality for a lot of spend.
At every setting, Levanto matched or beat OpenRouter Auto Beta on quality while costing 27–77% less. Levanto's quality dial also moves in the right direction: 8.3 points from quality-first to cost-first, where Auto Beta drops 23 points as you ask it to get cheaper.
| Setting | Levanto | OpenRouter Auto Beta | Savings |
|---|---|---|---|
| Quality-firstdial 0 | 98.3% · $0.0105 | 96.4% · $0.0260 | −60% |
| Balanceddial 5 | 98.2% · $0.0062 | 93.0% · $0.0264 | −77% |
| Cost-firstdial 10 | 90.0% · $0.0032 | 73.3% · $0.0044 | −27% |
Read before you cite this
What the benchmark does and doesn't show
- n = 60 AIME problems, point estimates only, no confidence intervals.
- OpenRouter Auto Beta ran live with an unrestricted model pool; Levanto used a three-model hull (Luna, Flash, Sol) with heads trained on multiple-choice data, not AIME.
- One model in the Levanto pool completed 52 of 60 keys, so the balanced setting averages the 55 of 60 picks that have scores.
- Levanto costs include the Sage routing fee; OpenRouter has no separate routing fee to add.
- Sage features were collected on the archive query text rather than the exact generation prompt.
One way to build it
How we built our router for this benchmark
Four steps. Yours will use different questions and a different pool, but the shape is the same.
- 01Profile
Batch closed-ended questions about the prompt in one Sage call.
- 02Vectorize
Each answer → value, confidence, value×confidence.
- 03Score
One small head per model: predicted quality and predicted cost.
- 04Pick
argmax(quality − λ·cost). λ is your quality–savings dial.
Our AIME benchmark used 10 questions (2 scales, 8 yes/no) → 30-d feature vector.
Dial λ from 0 (quality-first) to 10 (cost-first).
Build it yourself
# pip install levanto
from levanto import LevantoClient, Scale, YesNo
client = LevantoClient(api_key="lv_live_...")
# Ten closed-ended questions, one batch call: the prompt is billed once.
QUESTIONS = [
Scale("difficulty", "How much language-model reasoning capability is "
"intrinsically required to answer this query correctly?"),
Scale("knowledge_rarity", "How obscure or specialized is the knowledge needed?"),
YesNo("domain_math", "Is math a material capability needed?"),
YesNo("domain_science", "Is science a material capability needed?"),
YesNo("domain_code", "Is code a material capability needed?"),
YesNo("domain_logic_reasoning", "Is logic reasoning a material capability needed?"),
YesNo("skill_calculation", "Does this require non-trivial calculation?"),
YesNo("skill_code_synthesis", "Does this require writing or repairing executable code?"),
YesNo("skill_long_reasoning", "Does this require a long multi-step reasoning chain?"),
YesNo("skill_rare_knowledge", "Does this require obscure or specialized knowledge?"),
]
profile = client.decide_batch(prompt, QUESTIONS)Building a router? We'd like to compare notes.
We're refining this into something hosted. If you're routing between models today, or paying a frontier model to do a job a router should do, tell us what your pool and traffic look like.