Create cases with the API

Benchmarks is in preview and rolling out to organizations gradually. The endpoints and tools on this page are available once Benchmarks is enabled for your organization; until then they answer 403 not_entitled.

Everything the Add cases menu does in the dashboard, you can do over the REST API or MCP: write cases by hand, turn production conversations or traces into cases in bulk, and attach tool-call expectations and judge overrides. Both surfaces use the same service as the dashboard, so a case created over the API is identical to one created in the UI.

What a case is

A case is one input plus its expected outcome. Every case has a caseType:

`caseType`	Benchmark	Input	Expected outcome
`turn`	LLM	The conversation so far (messages, earlier tool calls and their results), the prompt to run against, and the call’s variables. The candidate produces one generation: a reply, tool calls, or both.	A reference reply, guidance on what it should have said instead, and/or a tool-call expectation.
`conversation`	LLM	A simulated caller (persona, goal, opening line), a turn limit, and tool mocks. The candidate talks to the caller until the conversation ends. Tools are mocked, never executed.	Rubric notes, whether the goal was reached, node path, and tools that must or must not be called.
`audio`	STT	An audio clip.	The reference transcript, plus optional key terms that must be transcribed exactly.

A turn case does not own a prompt. It points at an agent version (prompt.agentVersionId), so the same case can be replayed against several prompt revisions and models.

Authentication

Use an organization API key as a bearer token. The same key works for REST and MCP. Creating and editing cases is a write: the key must belong to an organization owner or admin, otherwise the request is refused with 403.

export ZELTO_API_KEY="zk_..."
export BENCHMARK_ID="..."   # from GET /v1/benchmarks or the benchmark's URL

Cases can only reference agents assigned to the benchmark. To add a case for another agent, assign that agent to the benchmark first.

Create cases

POST /v1/benchmarks/{id}/cases creates up to 100 cases per request and returns them in data.

curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/cases" \
  -H "Authorization: Bearer $ZELTO_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: import-2026-10-31-batch-1" \
  -d @cases.json

cases.json

{
  "cases": [
    {
      "caseType": "turn",
      "input": {
        "prompt": { "agentVersionId": "6f1c2a9e-..." },
        "variables": { "customer_name": "Jordan Reyes", "balance": "$240.00" },
        "messages": [
          { "role": "assistant", "content": "Hi, this is Ana from Acme. Am I speaking with Jordan?" },
          { "role": "user", "content": "Yes. I can't pay the full balance this month." }
        ]
      },
      "expected": {
        "replyExpectation": {
          "check": "different",
          "text": "Offer a payment plan before mentioning late fees.",
          "idealReply": "I understand. We can split the $240 into three monthly payments. Would that work?"
        },
        "mustNotContain": ["late fee"]
      },
      "tags": ["payment-plan", "hardship"]
    }
  ]
}

Response (201)

{
  "data": [
    {
      "id": "3f9a6c1e-...",
      "version": 1,
      "status": "ready",
      "source": "api",
      "sourceConversationId": null,
      "input": { "...": "..." },
      "expected": { "...": "..." },
      "groupId": null,
      "tags": ["payment-plan", "hardship"],
      "autoTags": ["source:api"]
    }
  ]
}

Send an Idempotency-Key header on batch creation so a retried request returns the cases it created the first time instead of creating duplicates.

Case fields

Field	Applies to	Description
`caseType`	all LLM cases	`turn` or `conversation`. STT cases omit it and are returned as `audio`.
`input.prompt`	`turn`, `conversation`	`{ "agentVersionId": "<uuid>" }`: the prompt the case runs against when a candidate uses the case’s own prompt.
`input.messages`	`turn`	The history before the step under test. Each message has a `role` (`user`, `assistant`, or `tool`) and `content`; a tool message carries `toolCall: { name, arguments, result }`.
`input.variables`	`turn`, `conversation`	Values for the prompt’s `{{variable}}` placeholders. A variable the prompt needs but the case doesn’t supply fails that result loudly instead of sending a raw placeholder.
`input.activeNodeId`	`turn`	For node-based agents: the node the conversation is in at this step.
`input.persona`	`conversation`	`{ role, goal, traits?, knowledge?, openingLine?, firstSpeaker? }` for the simulated caller. `firstSpeaker` is `agent` or `caller`.
`input.maxTurns`	`conversation`	Turn limit, up to 30 (default 12).
`input.toolMocks`	`conversation`	Canned results for tool calls: `{ tool, match?, response, once? }`. `match` uses the argument matchers.
`input.onUnmatchedTool`	`conversation`	`error` (default) fails the result when the agent calls a tool no mock matches; `generic_mock` returns `{"ok": true}`.
`input.startNodeId`, `input.prefix`	`conversation`	Optional start node, and optional messages to start from (for example the first turns of a real call).
`expected.referenceAnswer`	`turn`	The golden reply.
`expected.replyExpectation`	`turn`	How the reply is graded: `check` is `reference` (compare to the reference answer), `different` (graded against `text`, what the agent should have done, with an optional `idealReply`), or `none` (reply judges don’t apply to this case).
`expected.toolExpectation`	`turn`	What tool calls the step must or must not make. See Tool-call expectations.
`expected.rubricNotes`	LLM	Case-specific guidance passed to LLM Judges.
`expected.mustContain`, `expected.mustNotContain`	`turn`	Phrases the reply must include or avoid.
`expected.expectedTransition`	`turn`	For node-based agents: `{ "toNodeId": "..." }` or `{ "stay": true }`.
`expected.goalReached`, `pathIncludes`, `pathExcludes`, `toolCalled`, `toolNotCalled`	`conversation`	Conversation outcome checks.
`skippedJudgeIds`	all	Judges to skip for this case. See Skip or tune a judge.
`judgeParamOverrides`	all	Per-case values for a judge’s parameters.
`variableOverrides`	cases from production	Replace a captured variable for this case: `{ "<name>": { "value": "...", "kind": "edited" \| "placeholder" } }`. Only names the case captured are accepted.
`groupId`	all	The group the case belongs to (at most one).
`tags`	all	Free-form labels used to slice results. They never affect scoring.

A turn case must check at least one thing: a reply expectation other than none, or a tool expectation.

Tool-call expectations

toolExpectation checks whether the step called the right tool and whether the call has the right form. Pick one of three shapes:

`expect`	Passes when
`call`	The step makes the listed `calls` (in order if `ordered` is `true`) and none of the `forbiddenTools`.
`no_call`	The step answers directly and calls no tool.
`any`	The step may call tools or not, but never one of the `forbiddenTools`.

This case expects a payment plan to be created with three installments, in the customer’s currency, and never a refund:

{
  "expected": {
    "replyExpectation": { "check": "none" },
    "toolExpectation": {
      "expect": "call",
      "ordered": true,
      "calls": [
        {
          "tool": "create_payment_plan",
          "arguments": {
            "installments": { "op": "equals", "value": 3 },
            "currency": { "op": "one_of", "values": ["USD", "usd"] },
            "first_payment_date": { "op": "semantic", "description": "a date within the next 7 days" },
            "notes": { "op": "ignore" }
          }
        }
      ],
      "forbiddenTools": ["issue_refund"],
      "noExtraArguments": false
    }
  }
}

Only the arguments you list are checked; set noExtraArguments: true to fail calls that send any other argument. When a step both speaks and calls tools, replyAt (with ordered: true) pins where the reply sits among the calls: 0 means before every call. Every listed tool must exist in the case’s prompt.

Argument matchers

`op`	Passes when
`equals`	The value deep-equals `value`.
`one_of`	The value equals one of `values`.
`regex`	The value matches `pattern` (RE2 syntax).
`number_range`	The number is between `min` and `max` (either is optional).
`present`	Any non-null value.
`semantic`	An LLM judges that the value fits `description`, for example “means next Friday”. This is the only matcher with a judge cost.
`ignore`	Never checked.

Separately from matchers, the arguments of every call must validate against the tool’s JSON schema: "amount": "240" fails a { "type": "number" } argument.

Skip or tune a judge for one case

Judges are defined on the benchmark. Get their ids from GET /v1/benchmarks/{id} (validators[]); the API calls judges “validators”.

skippedJudgeIds opts this case out of judges that don’t make sense for it, such as a tone rubric on a tool-only step. Required judges can’t be skipped, and at least one judge must still run.
judgeParamOverrides changes a judge’s parameters for this case only. A judge labelled “Reply under words” with a default of 60 can be tightened to 10 for one case:

{
  "skippedJudgeIds": ["<tone-judge-id>"],
  "judgeParamOverrides": {
    "<length-judge-id>": { "maxWords": 10 }
  }
}

Every judge id and parameter key must exist on the benchmark, and values must match the parameter’s type and stay within its min / max. An override for a judge you also skip is dropped.

Conversation cases

{
  "caseType": "conversation",
  "input": {
    "prompt": { "agentVersionId": "6f1c2a9e-..." },
    "persona": {
      "role": "Customer who missed two payments",
      "goal": "Get the balance split into smaller payments",
      "traits": "Polite but firm; hangs up if pressured",
      "firstSpeaker": "agent"
    },
    "maxTurns": 12,
    "variables": { "customer_name": "Jordan Reyes", "balance": "$240.00" },
    "toolMocks": [
      { "tool": "get_balance", "response": { "balance": 240, "currency": "USD" } },
      {
        "tool": "create_payment_plan",
        "match": { "installments": { "op": "number_range", "min": 2, "max": 6 } },
        "response": { "ok": true, "planId": "pp_123" }
      }
    ],
    "onUnmatchedTool": "error"
  },
  "expected": {
    "rubricNotes": "The agent offers a plan before mentioning fees and confirms the first payment date.",
    "goalReached": true,
    "toolNotCalled": { "tool": "issue_refund", "arguments": {} }
  }
}

Audio cases (STT)

For an STT benchmark, each case is { audio, referenceTranscript, keyTerms?, tags? }. Audio never goes through the request body except for short clips:

Stage the file with POST /v1/recording-uploads, then PUT the bytes to the returned uploadUrl.
Create the case with "audio": { "uploadId": "<id>" }.

You can also pass "audio": { "url": "https://..." } (fetched by Zelto; the case is preparing until the download finishes) or, for clips up to 3 MB, "audio": { "base64": "...", "contentType": "audio/wav" }.

{
  "cases": [
    {
      "audio": { "uploadId": "<upload-id>" },
      "referenceTranscript": "My account number is 4 4 7 1 9, and I'd like to cancel.",
      "keyTerms": ["44719"],
      "tags": ["noisy", "lang:en"]
    }
  ]
}

Add cases from production calls

POST /v1/benchmarks/{id}/cases/from-conversations turns up to 100 conversations into cases in one request, using the same mapping as From production calls in the dashboard. Each case records the call’s agent version, the variable values captured from the call, the active node (for node-based agents), and the transcript before the evaluated turn.

curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/cases/from-conversations" \
  -H "Authorization: Bearer $ZELTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "conversationIds": ["0b6e...", "1c7f...", "2d80..."],
    "turn": "last_assistant"
  }'

Field	Description
`conversationIds`	Up to 100 conversation ids. Find them with `GET /v1/conversations`. Every call must belong to an agent assigned to the benchmark.
`turn`	LLM: `last_assistant` (default) or a turn number.
`as`	LLM: `turn` (default) or `conversation`. `conversation` drafts a persona, goal and tool mocks from each call and returns them as `draft` cases for you to review.
`trim`	STT: `{ "startSeconds": 12, "endSeconds": 31 }` to cut the recording to one utterance. The recording is copied into the benchmark at creation time, and the reference is prefilled from the call’s transcript and marked unverified.

Captured variables are masked in the response. Cases whose node couldn’t be resolved come back as draft. Use PATCH /v1/benchmarks/{id}/cases/{caseId} to refine any case afterwards: change its expected outcome, add a tool expectation, override a captured variable with variableOverrides, or set "status": "ready" to accept a draft.

Add cases from traces

POST /v1/benchmarks/{id}/cases/from-traces creates up to 100 cases from traces, the same as Add to benchmark on a trace or span.

curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/cases/from-traces" \
  -H "Authorization: Bearer $ZELTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "00f067aa0ba902b7" },
      { "traceId": "a1b2c3d4e5f60718293a4b5c6d7e8f90" }
    ]
  }'

With a spanId (an LLM or tool span), the case tests that one LLM call: the span’s own input messages become the history, and its output becomes the expected step.
Without a spanId, the trace’s linked conversation is used, like from-conversations with turn: "last_assistant".

Each created case reports turnSource: span_messages when the history came from the span, or transcript when it came from the linked call. A trace with no linked call needs a span that recorded its messages, otherwise that item fails with 422 no_turn_source.

Via MCP

The Zelto MCP server exposes the same operations as tools, so you can ask Claude, Cursor, or any MCP client to build a benchmark for you:

Tool	Does	Limit per call
`add_benchmark_cases`	Create `turn` or `conversation` cases (same shape as `POST …/cases`, plus `groupId` and `tags`).	50
`add_benchmark_cases_from_conversations`	Turn production calls into cases.	50
`add_benchmark_cases_from_traces`	Turn traces or spans into cases (`{ traceId, spanId? }` items).	50
`create_benchmark_audio_upload`	Get a presigned upload URL (and a ready `curl -T` line) for an audio case.	—
`update_benchmark_case`, `retire_benchmark_case`	Edit a case (a new revision in the draft) or retire it.	—
`list_benchmarks`, `get_benchmark`, `list_benchmark_cases`	Find benchmarks, their judges, and existing cases.	—
`publish_benchmark_version`	Publish the draft.	—

An example prompt:

In Zelto, find the benchmark "Collections next-turn" and add a case from
each of the last 30 failed calls for the Collections agent, using the last
assistant turn. For calls where the agent should have offered a payment plan,
expect a create_payment_plan call with installments between 2 and 6, and
never issue_refund. Tag them "payment-plan". Show me the draft before publishing.

The client calls list_benchmarks, list_conversations, add_benchmark_cases_from_conversations, then update_benchmark_case for the tool expectations.

Where new cases land

New and edited cases go to the benchmark’s draft, the working copy shown as “Draft changes” in the dashboard. Runs use the latest published version by default, so new cases are part of a run only after you publish:

curl -X POST "https://api.zelto.ai/v1/benchmarks/$BENCHMARK_ID/versions" \
  -H "Authorization: Bearer $ZELTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "note": "Add 30 payment-plan cases from October calls" }'

Publishing excludes cases still in draft status and warns you about them. Owners and admins can also run the draft directly; that run is labelled “draft run”. Cases are never deleted: editing one creates a new revision, and retiring one removes it from the next version while older runs keep their results.

Errors

Errors are JSON: { "error": "<code>", "details": ... }.

Status	When
`400`	The body is invalid. `details` lists the issues, for example: a tool in a tool expectation that isn’t in the case’s prompt, `expect: "call"` with no calls, an invalid regex, a case with nothing to check, an unknown or required judge in `skippedJudgeIds` (or skipping every judge), an unknown parameter or out-of-range value in `judgeParamOverrides`, a `variableOverrides` name the case didn’t capture, more cases than the per-request limit, or a case or conversation whose agent isn’t assigned to the benchmark (`agent_not_assigned`; assign the agent first).
`403`	The key’s owner isn’t an organization owner or admin, or Benchmarks isn’t enabled for your organization (`not_entitled`).
`404`	The benchmark, conversation, trace, or agent version isn’t in your organization.
`409`	A conflict with the benchmark’s current state, for example publishing with no draft changes (`no_draft_changes`).
`422 no_turn_source`	A trace with no linked call and no span messages to build a case from.

Over MCP the same refusals come back as tool results with isError: true; see Error results.

Benchmarks: cases, judges, and versions.
Agent versions: what prompt.agentVersionId points at.
MCP: connect a client and authenticate.

Get started

Core concepts

Integrations

Guides

Account & settings

Legal

Create cases with the API

What a case is

Authentication

Create cases

Case fields

Tool-call expectations

Argument matchers

Skip or tune a judge for one case

Conversation cases

Audio cases (STT)

Add cases from production calls

Add cases from traces

Via MCP

Where new cases land

Errors

​What a case is

​Authentication

​Create cases

​Case fields

​Tool-call expectations

​Argument matchers

​Skip or tune a judge for one case

​Conversation cases

​Audio cases (STT)

​Add cases from production calls

​Add cases from traces

​Via MCP

​Where new cases land

​Errors

​Related

What a case is

Authentication

Create cases

Case fields

Tool-call expectations

Argument matchers

Skip or tune a judge for one case

Conversation cases

Audio cases (STT)

Add cases from production calls

Add cases from traces

Via MCP

Where new cases land

Errors

Related