Skip to main content
Benchmarks is in preview and rolling out to organizations gradually. If you don’t see Benchmarks in your sidebar yet, it isn’t enabled for your organization.
A benchmark is a versioned set of cases plus the judges that score them. You run it against one or more candidates (a model, a prompt revision, or both) and read the results side by side, so one benchmark answers “which model?” and “which prompt revision?” on the same cases. A benchmark is either LLM (does the agent produce the right reply or tool call?) or STT (does the transcription engine get the words right?), and it is assigned to one or more agents.

​
Cases

A case is one input plus its expected outcome. There are three types: Add cases from production conversations and traces, by hand, by uploading audio, or programmatically. To create cases from your own code or from an AI client, see Create cases with the API.

​
Judges

Judges are defined once on the benchmark and applied to every case: LLM Judges (an LLM grades against a rubric), Code Judges (sandboxed TypeScript), and Built-in Judges (exact or regex match, contains, word error rate, and tool-call checks). A case can skip a non-required judge or override a judge’s parameters for that case only.

​
Versions

Every edit to cases, judges, or groups lands in the benchmark’s draft. Publish freezes the draft into the next version (v1, v2, …), and runs use the latest published version by default, so two runs of the same version are always comparable.
  • Simulations test one prompt against synthetic users; benchmarks compare several candidates on the same cases.
  • Experiments compare versions on live production traffic.
  • MCP lets Claude, Cursor, or another client build and run benchmarks for you.