> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zelto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Compare models and prompt revisions on the same set of cases, scored by the same judges, before you switch production.

> [!NOTE]
>
> Benchmarks is in preview and rolling out to organizations gradually. If you
> don't see **Benchmarks** in your sidebar yet, it isn't enabled for your
> organization.

A **benchmark** is a versioned set of **cases** plus the **judges** that score
them. You run it against one or more **candidates** (a model, a prompt
revision, or both) and read the results side by side, so one benchmark answers
"which model?" and "which prompt revision?" on the same cases.

A benchmark is either **LLM** (does the agent produce the right reply or tool
call?) or **STT** (does the transcription engine get the words right?), and it is
assigned to one or more [agents](https://docs.zelto.ai/docs/agents.md).

## Cases

A case is one input plus its expected outcome. There are three types:

| Type | What it tests |
| - | - |
| `turn` | One LLM generation: given the conversation so far, produce the next reply and/or tool call. |
| `conversation` | A simulated caller (persona, goal, opening line) talks to the candidate until the conversation ends. Tools are mocked, never executed. |
| `audio` | An audio clip and its reference transcript, for STT benchmarks. |

Add cases from production [conversations](https://docs.zelto.ai/docs/conversations.md) and
[traces](https://docs.zelto.ai/docs/traces.md), by hand, by uploading audio, or programmatically. To
create cases from your own code or from an AI client, see
[Create cases with the API](https://docs.zelto.ai/docs/benchmarks/cases-api.md).

## Judges

Judges are defined once on the benchmark and applied to every case: **LLM
Judges** (an LLM grades against a rubric), **Code Judges** (sandboxed
TypeScript), and **Built-in Judges** (exact or regex match, contains, word
error rate, and [tool-call](https://docs.zelto.ai/docs/tool-calls.md) checks). A case can skip a
non-required judge or override a judge's parameters for that case only.

## Versions

Every edit to cases, judges, or groups lands in the benchmark's **draft**.
**Publish** freezes the draft into the next version (v1, v2, …), and runs use
the latest published version by default, so two runs of the same version are
always comparable.

## Related

* [Simulations](https://docs.zelto.ai/docs/simulations.md) test one prompt against synthetic users;
  benchmarks compare several candidates on the same cases.
* [Experiments](https://docs.zelto.ai/docs/experiments.md) compare versions on live production traffic.
* [MCP](https://docs.zelto.ai/docs/mcp.md) lets Claude, Cursor, or another client build and run
  benchmarks for you.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.