> ## Documentation Index > Fetch the complete documentation index at: https://docs.zelto.ai/llms.txt > Use this file to discover all available pages before exploring further. # Create cases with the API > Add benchmark cases from your code over the REST API, or from Claude, Cursor, and other clients over MCP. > [!NOTE] > > [Benchmarks](https://docs.zelto.ai/docs/benchmarks.md) is in preview and rolling out to organizations > gradually. The endpoints and tools on this page are available once Benchmarks is > enabled for your organization; until then they answer `403 not_entitled`. Everything the **Add cases** menu does in the dashboard, you can do over the REST API or [MCP](https://docs.zelto.ai/docs/mcp.md): write cases by hand, turn production [conversations](https://docs.zelto.ai/docs/conversations.md) or [traces](https://docs.zelto.ai/docs/traces.md) into cases in bulk, and attach tool-call expectations and judge overrides. Both surfaces use the same service as the dashboard, so a case created over the API is identical to one created in the UI. ## What a case is A case is one input plus its expected outcome. Every case has a `caseType`: | `caseType` | Benchmark | Input | Expected outcome | | - | - | - | - | | `turn` | LLM | The conversation so far (messages, earlier tool calls and their results), the prompt to run against, and the call's variables. The candidate produces **one generation**: a reply, tool calls, or both. | A reference reply, guidance on what it should have said instead, and/or a [tool-call expectation](#tool-call-expectations). | | `conversation` | LLM | A simulated caller (persona, goal, opening line), a turn limit, and tool mocks. The candidate talks to the caller until the conversation ends. Tools are mocked, never executed. | Rubric notes, whether the goal was reached, node path, and tools that must or must not be called. | | `audio` | STT | An audio clip. | The reference transcript, plus optional key terms that must be transcribed exactly. | A `turn` case does not own a prompt. It points at an [agent version](https://docs.zelto.ai/docs/agent-versions.md) (`prompt.agentVersionId`), so the same case can be replayed against several prompt revisions and models. ## Authentication Use an organization [API key](https://docs.zelto.ai/docs/settings.md#api-keys) as a bearer token. The same key works for REST and [MCP](https://docs.zelto.ai/docs/mcp.md#authentication). Creating and editing cases is a write: the key must belong to an organization **owner or admin**, otherwise the request is refused with `403`. ```bash theme={null} export ZELTO_API_KEY="zk_..." export BENCHMARK_ID="..." # from GET /v1/benchmarks or the benchmark's URL ``` Cases can only reference agents assigned to the benchmark. To add a case for another agent, assign that agent to the benchmark first. ## Create cases `POST /v1/benchmarks/{id}/cases` creates up to **100 cases per request** and returns them in `data`. ```bash theme={null} curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/cases" \ -H "Authorization: Bearer $ZELTO_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: import-2026-10-31-batch-1" \ -d @cases.json ``` ```json title="cases.json" theme={null} { "cases": [ { "caseType": "turn", "input": { "prompt": { "agentVersionId": "6f1c2a9e-..." }, "variables": { "customer_name": "Jordan Reyes", "balance": "$240.00" }, "messages": [ { "role": "assistant", "content": "Hi, this is Ana from Acme. Am I speaking with Jordan?" }, { "role": "user", "content": "Yes. I can't pay the full balance this month." } ] }, "expected": { "replyExpectation": { "check": "different", "text": "Offer a payment plan before mentioning late fees.", "idealReply": "I understand. We can split the $240 into three monthly payments. Would that work?" }, "mustNotContain": ["late fee"] }, "tags": ["payment-plan", "hardship"] } ] } ``` ```json Response (201) theme={null} { "data": [ { "id": "3f9a6c1e-...", "version": 1, "status": "ready", "source": "api", "sourceConversationId": null, "input": { "...": "..." }, "expected": { "...": "..." }, "groupId": null, "tags": ["payment-plan", "hardship"], "autoTags": ["source:api"] } ] } ``` Send an `Idempotency-Key` header on batch creation so a retried request returns the cases it created the first time instead of creating duplicates. ### Case fields | Field | Applies to | Description | | - | - | - | | `caseType` | all LLM cases | `turn` or `conversation`. STT cases omit it and are returned as `audio`. | | `input.prompt` | `turn`, `conversation` | `{ "agentVersionId": "" }`: the prompt the case runs against when a candidate uses the case's own prompt. | | `input.messages` | `turn` | The history before the step under test. Each message has a `role` (`user`, `assistant`, or `tool`) and `content`; a tool message carries `toolCall: { name, arguments, result }`. | | `input.variables` | `turn`, `conversation` | Values for the prompt's `{{variable}}` placeholders. A variable the prompt needs but the case doesn't supply fails that result loudly instead of sending a raw placeholder. | | `input.activeNodeId` | `turn` | For node-based agents: the node the conversation is in at this step. | | `input.persona` | `conversation` | `{ role, goal, traits?, knowledge?, openingLine?, firstSpeaker? }` for the simulated caller. `firstSpeaker` is `agent` or `caller`. | | `input.maxTurns` | `conversation` | Turn limit, up to 30 (default 12). | | `input.toolMocks` | `conversation` | Canned results for tool calls: `{ tool, match?, response, once? }`. `match` uses the [argument matchers](#argument-matchers). | | `input.onUnmatchedTool` | `conversation` | `error` (default) fails the result when the agent calls a tool no mock matches; `generic_mock` returns `{"ok": true}`. | | `input.startNodeId`, `input.prefix` | `conversation` | Optional start node, and optional messages to start from (for example the first turns of a real call). | | `expected.referenceAnswer` | `turn` | The golden reply. | | `expected.replyExpectation` | `turn` | How the reply is graded: `check` is `reference` (compare to the reference answer), `different` (graded against `text`, what the agent should have done, with an optional `idealReply`), or `none` (reply judges don't apply to this case). | | `expected.toolExpectation` | `turn` | What tool calls the step must or must not make. See [Tool-call expectations](#tool-call-expectations). | | `expected.rubricNotes` | LLM | Case-specific guidance passed to LLM Judges. | | `expected.mustContain`, `expected.mustNotContain` | `turn` | Phrases the reply must include or avoid. | | `expected.expectedTransition` | `turn` | For node-based agents: `{ "toNodeId": "..." }` or `{ "stay": true }`. | | `expected.goalReached`, `pathIncludes`, `pathExcludes`, `toolCalled`, `toolNotCalled` | `conversation` | Conversation outcome checks. | | `skippedJudgeIds` | all | Judges to skip for this case. See [Skip or tune a judge](#skip-or-tune-a-judge-for-one-case). | | `judgeParamOverrides` | all | Per-case values for a judge's parameters. | | `variableOverrides` | cases from production | Replace a captured variable for this case: `{ "": { "value": "...", "kind": "edited" \| "placeholder" } }`. Only names the case captured are accepted. | | `groupId` | all | The group the case belongs to (at most one). | | `tags` | all | Free-form labels used to slice results. They never affect scoring. | A `turn` case must check at least one thing: a reply expectation other than `none`, or a tool expectation. ### Tool-call expectations `toolExpectation` checks whether the step called the right tool and whether the call has the right form. Pick one of three shapes: | `expect` | Passes when | | - | - | | `call` | The step makes the listed `calls` (in order if `ordered` is `true`) and none of the `forbiddenTools`. | | `no_call` | The step answers directly and calls no tool. | | `any` | The step may call tools or not, but never one of the `forbiddenTools`. | This case expects a payment plan to be created with three installments, in the customer's currency, and never a refund: ```json theme={null} { "expected": { "replyExpectation": { "check": "none" }, "toolExpectation": { "expect": "call", "ordered": true, "calls": [ { "tool": "create_payment_plan", "arguments": { "installments": { "op": "equals", "value": 3 }, "currency": { "op": "one_of", "values": ["USD", "usd"] }, "first_payment_date": { "op": "semantic", "description": "a date within the next 7 days" }, "notes": { "op": "ignore" } } } ], "forbiddenTools": ["issue_refund"], "noExtraArguments": false } } } ``` Only the arguments you list are checked; set `noExtraArguments: true` to fail calls that send any other argument. When a step both speaks and calls tools, `replyAt` (with `ordered: true`) pins where the reply sits among the calls: `0` means before every call. Every listed `tool` must exist in the case's prompt. #### Argument matchers | `op` | Passes when | | - | - | | `equals` | The value deep-equals `value`. | | `one_of` | The value equals one of `values`. | | `regex` | The value matches `pattern` (RE2 syntax). | | `number_range` | The number is between `min` and `max` (either is optional). | | `present` | Any non-null value. | | `semantic` | An LLM judges that the value fits `description`, for example "means next Friday". This is the only matcher with a judge cost. | | `ignore` | Never checked. | Separately from matchers, the arguments of every call must validate against the tool's JSON schema: `"amount": "240"` fails a `{ "type": "number" }` argument. ### Skip or tune a judge for one case Judges are defined on the benchmark. Get their ids from `GET /v1/benchmarks/{id}` (`validators[]`); the API calls judges "validators". * **`skippedJudgeIds`** opts this case out of judges that don't make sense for it, such as a tone rubric on a tool-only step. Required judges can't be skipped, and at least one judge must still run. * **`judgeParamOverrides`** changes a judge's parameters for this case only. A judge labelled "Reply under {maxWords} words" with a default of 60 can be tightened to 10 for one case: ```json theme={null} { "skippedJudgeIds": [""], "judgeParamOverrides": { "": { "maxWords": 10 } } } ``` Every judge id and parameter key must exist on the benchmark, and values must match the parameter's type and stay within its `min` / `max`. An override for a judge you also skip is dropped. ### Conversation cases ```json theme={null} { "caseType": "conversation", "input": { "prompt": { "agentVersionId": "6f1c2a9e-..." }, "persona": { "role": "Customer who missed two payments", "goal": "Get the balance split into smaller payments", "traits": "Polite but firm; hangs up if pressured", "firstSpeaker": "agent" }, "maxTurns": 12, "variables": { "customer_name": "Jordan Reyes", "balance": "$240.00" }, "toolMocks": [ { "tool": "get_balance", "response": { "balance": 240, "currency": "USD" } }, { "tool": "create_payment_plan", "match": { "installments": { "op": "number_range", "min": 2, "max": 6 } }, "response": { "ok": true, "planId": "pp_123" } } ], "onUnmatchedTool": "error" }, "expected": { "rubricNotes": "The agent offers a plan before mentioning fees and confirms the first payment date.", "goalReached": true, "toolNotCalled": { "tool": "issue_refund", "arguments": {} } } } ``` ### Audio cases (STT) For an STT benchmark, each case is `{ audio, referenceTranscript, keyTerms?, tags? }`. Audio never goes through the request body except for short clips: 1. Stage the file with [`POST /v1/recording-uploads`](https://docs.zelto.ai/api-reference/conversations/create-a-direct-recording-upload), then `PUT` the bytes to the returned `uploadUrl`. 2. Create the case with `"audio": { "uploadId": "" }`. You can also pass `"audio": { "url": "https://..." }` (fetched by Zelto; the case is `preparing` until the download finishes) or, for clips up to 3 MB, `"audio": { "base64": "...", "contentType": "audio/wav" }`. ```json theme={null} { "cases": [ { "audio": { "uploadId": "" }, "referenceTranscript": "My account number is 4 4 7 1 9, and I'd like to cancel.", "keyTerms": ["44719"], "tags": ["noisy", "lang:en"] } ] } ``` ## Add cases from production calls `POST /v1/benchmarks/{id}/cases/from-conversations` turns up to **100** [conversations](https://docs.zelto.ai/docs/conversations.md) into cases in one request, using the same mapping as **From production calls** in the dashboard. Each case records the call's agent version, the variable values captured from the call, the active node (for node-based agents), and the transcript before the evaluated turn. ```bash theme={null} curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/cases/from-conversations" \ -H "Authorization: Bearer $ZELTO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "conversationIds": ["0b6e...", "1c7f...", "2d80..."], "turn": "last_assistant" }' ``` | Field | Description | | - | - | | `conversationIds` | Up to 100 conversation ids. Find them with [`GET /v1/conversations`](https://docs.zelto.ai/api-reference/conversations/list-conversations). Every call must belong to an agent assigned to the benchmark. | | `turn` | LLM: `last_assistant` (default) or a turn number. | | `as` | LLM: `turn` (default) or `conversation`. `conversation` drafts a persona, goal and tool mocks from each call and returns them as `draft` cases for you to review. | | `trim` | STT: `{ "startSeconds": 12, "endSeconds": 31 }` to cut the recording to one utterance. The recording is copied into the benchmark at creation time, and the reference is prefilled from the call's transcript and marked unverified. | Captured variables are masked in the response. Cases whose node couldn't be resolved come back as `draft`. Use `PATCH /v1/benchmarks/{id}/cases/{caseId}` to refine any case afterwards: change its expected outcome, add a tool expectation, override a captured variable with `variableOverrides`, or set `"status": "ready"` to accept a draft. ## Add cases from traces `POST /v1/benchmarks/{id}/cases/from-traces` creates up to **100** cases from [traces](https://docs.zelto.ai/docs/traces.md), the same as **Add to benchmark** on a trace or span. ```bash theme={null} curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/cases/from-traces" \ -H "Authorization: Bearer $ZELTO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "items": [ { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "00f067aa0ba902b7" }, { "traceId": "a1b2c3d4e5f60718293a4b5c6d7e8f90" } ] }' ``` * **With a `spanId`** (an LLM or tool span), the case tests that one LLM call: the span's own input messages become the history, and its output becomes the expected step. * **Without a `spanId`**, the trace's [linked conversation](https://docs.zelto.ai/docs/traces.md#link-traces-to-conversations) is used, like `from-conversations` with `turn: "last_assistant"`. Each created case reports `turnSource`: `span_messages` when the history came from the span, or `transcript` when it came from the linked call. A trace with no linked call needs a span that recorded its messages, otherwise that item fails with `422 no_turn_source`. ## Via MCP The [Zelto MCP server](https://docs.zelto.ai/docs/mcp.md) exposes the same operations as tools, so you can ask Claude, Cursor, or any MCP client to build a benchmark for you: | Tool | Does | Limit per call | | - | - | - | | `add_benchmark_cases` | Create `turn` or `conversation` cases (same shape as `POST …/cases`, plus `groupId` and `tags`). | 50 | | `add_benchmark_cases_from_conversations` | Turn production calls into cases. | 50 | | `add_benchmark_cases_from_traces` | Turn traces or spans into cases (`{ traceId, spanId? }` items). | 50 | | `create_benchmark_audio_upload` | Get a presigned upload URL (and a ready `curl -T` line) for an audio case. | — | | `update_benchmark_case`, `retire_benchmark_case` | Edit a case (a new revision in the draft) or retire it. | — | | `list_benchmarks`, `get_benchmark`, `list_benchmark_cases` | Find benchmarks, their judges, and existing cases. | — | | `publish_benchmark_version` | Publish the draft. | — | An example prompt: ```text theme={null} In Zelto, find the benchmark "Collections next-turn" and add a case from each of the last 30 failed calls for the Collections agent, using the last assistant turn. For calls where the agent should have offered a payment plan, expect a create_payment_plan call with installments between 2 and 6, and never issue_refund. Tag them "payment-plan". Show me the draft before publishing. ``` The client calls `list_benchmarks`, `list_conversations`, `add_benchmark_cases_from_conversations`, then `update_benchmark_case` for the tool expectations. ## Where new cases land New and edited cases go to the benchmark's **draft**, the working copy shown as "Draft changes" in the dashboard. Runs use the latest **published** version by default, so new cases are part of a run only after you publish: ```bash theme={null} curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/versions" \ -H "Authorization: Bearer $ZELTO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "note": "Add 30 payment-plan cases from October calls" }' ``` Publishing excludes cases still in `draft` status and warns you about them. Owners and admins can also run the draft directly; that run is labelled "draft run". Cases are never deleted: editing one creates a new revision, and retiring one removes it from the next version while older runs keep their results. ## Errors Errors are JSON: `{ "error": "", "details": ... }`. | Status | When | | - | - | | `400` | The body is invalid. `details` lists the issues, for example: a tool in a tool expectation that isn't in the case's prompt, `expect: "call"` with no calls, an invalid regex, a case with nothing to check, an unknown or required judge in `skippedJudgeIds` (or skipping every judge), an unknown parameter or out-of-range value in `judgeParamOverrides`, a `variableOverrides` name the case didn't capture, more cases than the per-request limit, or a case or conversation whose agent isn't assigned to the benchmark (`agent_not_assigned`; assign the agent first). | | `403` | The key's owner isn't an organization owner or admin, or Benchmarks isn't enabled for your organization (`not_entitled`). | | `404` | The benchmark, conversation, trace, or agent version isn't in your organization. | | `409` | A conflict with the benchmark's current state, for example publishing with no draft changes (`no_draft_changes`). | | `422 no_turn_source` | A trace with no linked call and no span messages to build a case from. | Over MCP the same refusals come back as tool results with `isError: true`; see [Error results](https://docs.zelto.ai/docs/mcp.md#error-results). ## Related * [Benchmarks](https://docs.zelto.ai/docs/benchmarks.md): cases, judges, and versions. * [Agent versions](https://docs.zelto.ai/docs/agent-versions.md): what `prompt.agentVersionId` points at. * [MCP](https://docs.zelto.ai/docs/mcp.md): connect a client and authenticate. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.