Optimize context

Give your LLM only the documents a question needs.

Stuffing every document into the prompt is the easy way to give an LLM context. It's also how a question about a broken grinder ends up reading the API reference and the notes from a 2019 offsite.

Ask Sage2 which documents the question needs first, with one Tags call, and send only those. The LLM reads fewer tokens, and fewer irrelevant pages pull its answer off course.

Tags

Try

Incoming question
```
My grinder broke after 8 months. Can I get my money back or a free repair?
```
My grinder broke after 8 months. Can I get my money back or a free repair?
Documents the LLM could read
- Refund policy: How refunds work, time limits, and exceptions (640 tokens)
- Shipping rates: Delivery prices and times by country (820 tokens)
- API reference: Endpoints and authentication for developers (6,200 tokens)
- Warranty terms: What is covered when a product breaks, and for how long (910 tokens)
- Brand voice guide: How we write marketing copy (1,450 tokens)
- 2019 offsite notes: Team retreat agenda and who brought the guitar (2,300 tokens)
Mark every document the question needsTags
- refund_policyapplies
- shipping_ratesno
- api_referenceno
- warrantyapplies
- brand_voiceno
- offsite_2019no
161 ms
Your code2 of 6 documents sent
The LLM reads 1,550 of 12,320 tokens.

One Tags question covered all 6 documents in one request: 161 ms.

Get the code

Set SAGE_API_KEY (get a free key), then run it. Python: pip install requests. TypeScript: Node 22+. Save as recipe.mts and run npx tsx recipe.mts.

A null answer is not an error. Sage2 isn't sure, and each step below says what to do with it. Limits, errors and retries: Limits and errors.

Markdown

import os
import requests

def sage(path, body):
    r = requests.post(
        "https://sage.levanto.ai" + path,
        headers={"Authorization": f"Bearer {os.environ['SAGE_API_KEY']}"},
        json=body,
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

question = "My grinder broke after 8 months. Can I get my money back or a free repair?"

DOCS = [
    {
        "id": "refund_policy",
        "title": "Refund policy",
        "summary": "How refunds work, time limits, and exceptions",
    },
    {
        "id": "shipping_rates",
        "title": "Shipping rates",
        "summary": "Delivery prices and times by country",
    },
    {
        "id": "api_reference",
        "title": "API reference",
        "summary": "Endpoints and authentication for developers",
    },
    {
        "id": "warranty",
        "title": "Warranty terms",
        "summary": "What is covered when a product breaks, and for how long",
    },
    {
        "id": "brand_voice",
        "title": "Brand voice guide",
        "summary": "How we write marketing copy",
    },
    {
        "id": "offsite_2019",
        "title": "2019 offsite notes",
        "summary": "Team retreat agenda and who brought the guitar",
    },
]

tags = sage("/decide", {
    "content": question,
    "question": {
        "id": "context", "kind": "tags",
        "instructions": "An assistant has to answer this question. Tag each document it needs to read. A document it doesn't need does not apply.",
        "tags": [{"id": doc["id"], "name": f"{doc['title']}: {doc['summary']}"} for doc in DOCS],
    },
})["result"]["tags"]

# Keep what applies, and what Sage isn't sure about (applies is None).
needed = [t["id"] for t in tags if t["applies"] is not False]
context = [doc for doc in DOCS if doc["id"] in needed]
print("Give the LLM:", [doc["title"] for doc in context])

How it works

Your code

Describe your documents

Give each document an id, a title and a one-line summary. Sage2 decides from the summary alone, never the full document.

Runs with the helper from Get the code.

question = "My grinder broke after 8 months. Can I get my money back or a free repair?"

DOCS = [
    {
        "id": "refund_policy",
        "title": "Refund policy",
        "summary": "How refunds work, time limits, and exceptions",
    },
    {
        "id": "shipping_rates",
        "title": "Shipping rates",
        "summary": "Delivery prices and times by country",
    },
    {
        "id": "api_reference",
        "title": "API reference",
        "summary": "Endpoints and authentication for developers",
    },
    {
        "id": "warranty",
        "title": "Warranty terms",
        "summary": "What is covered when a product breaks, and for how long",
    },
    {
        "id": "brand_voice",
        "title": "Brand voice guide",
        "summary": "How we write marketing copy",
    },
    {
        "id": "offsite_2019",
        "title": "2019 offsite notes",
        "summary": "Team retreat agenda and who brought the guitar",
    },
]

Ask which ones the question needs

Tags docs

Send the user's question as content and one tag per document. Each tag's name is what Sage2 reads. Its id is the key you get back.

For every tag, Sage2 returns applies (true, false, or null when it's too close to call) and a probability.

Runs with the helper from Get the code.

tags = sage("/decide", {
    "content": question,
    "question": {
        "id": "context", "kind": "tags",
        "instructions": "An assistant has to answer this question. Tag each document it needs to read. A document it doesn't need does not apply.",
        "tags": [{"id": doc["id"], "name": f"{doc['title']}: {doc['summary']}"} for doc in DOCS],
    },
})["result"]["tags"]

Example Sage2 output

{
  "id": "context",
  "kind": "tags",
  "result": {
    "tags": [
      {
        "id": "refund_policy",
        "probability": 0.9984,
        "applies": true
      },
      {
        "id": "shipping_rates",
        "probability": 0,
        "applies": false
      },
      {
        "id": "api_reference",
        "probability": 0,
        "applies": false
      },
      {
        "id": "warranty",
        "probability": 1,
        "applies": true
      },
      {
        "id": "brand_voice",
        "probability": 0,
        "applies": false
      },
      {
        "id": "offsite_2019",
        "probability": 0,
        "applies": false
      }
    ]
  },
  "meta": {
    "model": "levanto-sage-2",
    "latency_ms": 160.6,
    "question_count": 1,
    "usage": {
      "billed_input_tokens": 1078,
      "image_count": 0,
      "image_tokens": 0,
      "input_tokens": 1078,
      "output_tokens": 0
    }
  }
}

Your code

Build the prompt

Send your LLM the documents that apply. Keep the unsure ones too (applies is null). One extra page costs less than a wrong answer.

Runs with the helper from Get the code.

# Keep what applies, and what Sage isn't sure about (applies is None).
needed = [t["id"] for t in tags if t["applies"] is not False]
context = [doc for doc in DOCS if doc["id"] in needed]
print("Give the LLM:", [doc["title"] for doc in context])

Before you ship

Summaries do the work. If Sage2 picks a document it should not, sharpen the summary of that document.
Check permissions in code. Sage2 picks what's relevant, not what this user may see. List only the documents they may read.
Use Tags here. A question can need several documents. Each document gets its own verdict.
Up to 120 tags per question. For a larger library, narrow it down first, for example by product or section.
Cost grows with the number of documents, not their length. Sage2 reads only the summaries. Six documents take about 0.1 s and 1,100 tokens.