# Triage bug screenshots

Check that a screenshot shows the reported bug and send it to the right team. Unclear ones go to a person.

Category: Read documents. Live demo: https://levanto.ai/examples/screenshot-classification

"It's broken" plus a screenshot is a common bug report, and someone has to look before anyone can fix it. This guide sends the screenshot and the ticket to Sage2 and asks two things at once: **does the image show the problem**, and **which team should take it**.

## Setup

Base URL: `https://sage.levanto.ai`. Send `Authorization: Bearer $SAGE_API_KEY` on every call. Get a key, free, at https://platform.levanto.ai. Keep the key on your server.

A `null` answer is not an error: Sage2 isn't sure. Each step says what to do with it.

## Step by step

### 1. Send the screenshot with the ticket

Image content is an object with `kind: "image"`, the picture as a base64 data URL in `media`, and optional `text`. Put the customer's ticket in `text`. Sage2 then judges the image against what the customer said.

Images are in beta. Send PNG, JPEG or WebP, up to 4 MiB and 8,192 px on the longest side. Image tokens are billed as input tokens.

**Using curl?** The curl tab shortens the image in `media`. Replace everything after `base64,` with your own image. `base64 -i screenshot.png | tr -d '\n'` prints it.

```python
import base64

path = "screenshot.png"  # PNG, JPEG or WebP
fmt = path.rsplit(".", 1)[-1].lower().replace("jpg", "jpeg")
with open(path, "rb") as f:
    media = f"data:image/{fmt};base64," + base64.b64encode(f.read()).decode()

ticket = "The pay button is missing on my phone."
```

```typescript
import { readFileSync } from "node:fs";

const path = "screenshot.png"; // PNG, JPEG or WebP
const fmt = path.split(".").pop()!.toLowerCase().replace("jpg", "jpeg");
const media = `data:image/${fmt};base64,` + readFileSync(path).toString("base64");

const ticket = "The pay button is missing on my phone.";
```

### 2. Ask both questions in one call

`/decide/batch` takes a list of `requests`. Each has one `content` and several `questions`, here:

- `shows_problem` is a Yes/No question: is the bug on screen?
- `team` is a Choice question between frontend, backend and support.

Both answers come back in one response.

```python
out = sage("/decide/batch", {
    "requests": [
        {
            "content": {
                "kind": "image",
                "media": media,
                "text": f"Ticket: {ticket}",
            },
            "questions": [
                {
                    "id": "shows_problem",
                    "kind": "yesno",
                    "instructions": "Does the screenshot show the problem described in the ticket?",
                },
                {
                    "id": "team",
                    "kind": "choice",
                    "instructions": "Which team should look at this screenshot?",
                    "options": [
                        {
                            "option": "frontend",
                            "description": "A layout problem, or something missing from the page",
                        },
                        {
                            "option": "backend",
                            "description": "A server or payment error message",
                        },
                        {
                            "option": "support",
                            "description": "The page looks correct, so ask the customer for more detail",
                        },
                    ],
                },
            ],
        },
    ],
})
```

```typescript
const out = await sage("/decide/batch", {
  requests: [
    {
      content: {
        kind: "image",
        media: media,
        text: `Ticket: ${ticket}`,
      },
      questions: [
        {
          id: "shows_problem",
          kind: "yesno",
          instructions: "Does the screenshot show the problem described in the ticket?",
        },
        {
          id: "team",
          kind: "choice",
          instructions: "Which team should look at this screenshot?",
          options: [
            {
              option: "frontend",
              description: "A layout problem, or something missing from the page",
            },
            { option: "backend", description: "A server or payment error message" },
            {
              option: "support",
              description: "The page looks correct, so ask the customer for more detail",
            },
          ],
        },
      ],
    },
  ],
});
```

### 3. Route the ticket

Open a ticket only when the screenshot supports the report. If it does not, ask the customer for more detail. If either answer is `null`, a person looks at it.

```python
def answers(out, group=0):
    # Each answer is a full decision. Failed ones (ok is false) are left out: treat a missing id as "not sure".
    return {a["result"]["id"]: a["result"]["result"] for a in out["results"][group]["answers"] if a["ok"]}

a = answers(out)
shows = a.get("shows_problem", {}).get("answer")  # a missing answer reads as not sure
team = a.get("team", {}).get("chosen")
if shows is None or team is None:
    print("Not sure: a person takes a look")
elif shows == "no" or team == "support":
    print("Ask the customer for more detail")
else:
    print("Open a ticket for the", team, "team")
```

```typescript
// Each answer is a full decision. Failed ones (ok is false) are left out: treat a missing id as "not sure".
const answers = (out: any, group = 0): Record<string, any> =>
  Object.fromEntries(out.results[group].answers.filter((a: any) => a.ok).map((a: any) => [a.result.id, a.result.result]));

const a = answers(out);
const shows: string | null = a.shows_problem?.answer ?? null; // a missing answer reads as not sure
const team: string | null = a.team?.chosen ?? null;
if (shows === null || team === null) {
  console.log("Not sure: a person takes a look");
} else if (shows === "no" || team === "support") {
  console.log("Ask the customer for more detail");
} else {
  console.log("Open a ticket for the", team, "team");
}
```

## Before you ship

- **Scale screenshots down** before you send them. Image tokens are billed, so a smaller picture costs less.
- **A `no` is useful.** A `no` on `shows_problem` stops an engineer from chasing a bug that is not there.
- **Set your own bar.** Each answer has a `probability`. Test on a few hundred past tickets, then assign a team automatically only above the level you trust.
- **Fast and small.** One call answers both questions in about 0.25 s. A phone screenshot scaled to 1,200 px reads about 700 input tokens, most of them for the image.

## Full recipe (Python, `pip install requests`)

```python
import os
import requests

def sage(path, body):
    r = requests.post(
        "https://sage.levanto.ai" + path,
        headers={"Authorization": f"Bearer {os.environ['SAGE_API_KEY']}"},
        json=body,
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

import base64

path = "screenshot.png"  # PNG, JPEG or WebP
fmt = path.rsplit(".", 1)[-1].lower().replace("jpg", "jpeg")
with open(path, "rb") as f:
    media = f"data:image/{fmt};base64," + base64.b64encode(f.read()).decode()

ticket = "The pay button is missing on my phone."

out = sage("/decide/batch", {
    "requests": [
        {
            "content": {
                "kind": "image",
                "media": media,
                "text": f"Ticket: {ticket}",
            },
            "questions": [
                {
                    "id": "shows_problem",
                    "kind": "yesno",
                    "instructions": "Does the screenshot show the problem described in the ticket?",
                },
                {
                    "id": "team",
                    "kind": "choice",
                    "instructions": "Which team should look at this screenshot?",
                    "options": [
                        {
                            "option": "frontend",
                            "description": "A layout problem, or something missing from the page",
                        },
                        {
                            "option": "backend",
                            "description": "A server or payment error message",
                        },
                        {
                            "option": "support",
                            "description": "The page looks correct, so ask the customer for more detail",
                        },
                    ],
                },
            ],
        },
    ],
})

def answers(out, group=0):
    # Each answer is a full decision. Failed ones (ok is false) are left out: treat a missing id as "not sure".
    return {a["result"]["id"]: a["result"]["result"] for a in out["results"][group]["answers"] if a["ok"]}

a = answers(out)
shows = a.get("shows_problem", {}).get("answer")  # a missing answer reads as not sure
team = a.get("team", {}).get("chosen")
if shows is None or team is None:
    print("Not sure: a person takes a look")
elif shows == "no" or team == "support":
    print("Ask the customer for more detail")
else:
    print("Open a ticket for the", team, "team")
```

## Full recipe (TypeScript, Node 22+, nothing to install: save as `recipe.mts`, run `npx tsx recipe.mts`)

```typescript
// Node 22+. Save as recipe.mts and run npx tsx recipe.mts
async function sage(path: string, body: unknown): Promise<any> {
  const r = await fetch("https://sage.levanto.ai" + path, {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.SAGE_API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(body),
    signal: AbortSignal.timeout(30_000),
  });
  if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
  return r.json();
}

import { readFileSync } from "node:fs";

const path = "screenshot.png"; // PNG, JPEG or WebP
const fmt = path.split(".").pop()!.toLowerCase().replace("jpg", "jpeg");
const media = `data:image/${fmt};base64,` + readFileSync(path).toString("base64");

const ticket = "The pay button is missing on my phone.";

const out = await sage("/decide/batch", {
  requests: [
    {
      content: {
        kind: "image",
        media: media,
        text: `Ticket: ${ticket}`,
      },
      questions: [
        {
          id: "shows_problem",
          kind: "yesno",
          instructions: "Does the screenshot show the problem described in the ticket?",
        },
        {
          id: "team",
          kind: "choice",
          instructions: "Which team should look at this screenshot?",
          options: [
            {
              option: "frontend",
              description: "A layout problem, or something missing from the page",
            },
            { option: "backend", description: "A server or payment error message" },
            {
              option: "support",
              description: "The page looks correct, so ask the customer for more detail",
            },
          ],
        },
      ],
    },
  ],
});

// Each answer is a full decision. Failed ones (ok is false) are left out: treat a missing id as "not sure".
const answers = (out: any, group = 0): Record<string, any> =>
  Object.fromEntries(out.results[group].answers.filter((a: any) => a.ok).map((a: any) => [a.result.id, a.result.result]));

const a = answers(out);
const shows: string | null = a.shows_problem?.answer ?? null; // a missing answer reads as not sure
const team: string | null = a.team?.chosen ?? null;
if (shows === null || team === null) {
  console.log("Not sure: a person takes a look");
} else if (shows === "no" || team === "support") {
  console.log("Ask the customer for more detail");
} else {
  console.log("Open a ticket for the", team, "team");
}
```

## Requests

### 1. POST /decide/batch

```bash
curl -s https://sage.levanto.ai/decide/batch \
  -H "Authorization: Bearer $SAGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<'JSON'
{
  "requests": [
    {
      "content": {
        "kind": "image",
        "media": "data:image/png;base64,iVBORw0KGgo...",
        "text": "Ticket: The pay button is missing on my phone."
      },
      "questions": [
        {
          "id": "shows_problem",
          "kind": "yesno",
          "instructions": "Does the screenshot show the problem described in the ticket?"
        },
        {
          "id": "team",
          "kind": "choice",
          "instructions": "Which team should look at this screenshot?",
          "options": [
            {
              "option": "frontend",
              "description": "A layout problem, or something missing from the page"
            },
            {
              "option": "backend",
              "description": "A server or payment error message"
            },
            {
              "option": "support",
              "description": "The page looks correct, so ask the customer for more detail"
            }
          ]
        }
      ]
    }
  ]
}
JSON
```

Response, abridged (a real run):

```json
{
  "results": [
    {
      "answers": [
        {
          "ok": true,
          "result": {
            "id": "shows_problem",
            "kind": "yesno",
            "result": {
              "answer": "yes",
              "probability": 0.946
            },
            "meta": {
              "model": "levanto-sage-2"
            }
          },
          "error": null
        },
        {
          "ok": true,
          "result": {
            "id": "team",
            "kind": "choice",
            "result": {
              "chosen": "frontend",
              "probability": 0.9486,
              "probabilities": [
                {
                  "option": "frontend",
                  "probability": 0.9486
                },
                {
                  "option": "backend",
                  "probability": 0.0092
                },
                {
                  "option": "support",
                  "probability": 0.0422
                }
              ]
            },
            "meta": {
              "model": "levanto-sage-2"
            }
          },
          "error": null
        }
      ]
    }
  ],
  "meta": {
    "model": "levanto-sage-2",
    "request_count": 1,
    "question_count": 2,
    "latency_ms": 185.3,
    "usage": {
      "billed_input_tokens": 716,
      "image_count": 1,
      "image_tokens": 541,
      "input_tokens": 175,
      "output_tokens": 0
    }
  }
}
```

## Docs

- Images: https://docs.levanto.ai/decision-model/images
- Batch: https://docs.levanto.ai/decision-model/batch
- Limits and errors: https://docs.levanto.ai/decision-model/limits
- OpenAPI: https://sage.levanto.ai/openapi.json