> ## Documentation Index > Fetch the complete documentation index at: https://docs.zelto.ai/llms.txt > Use this file to discover all available pages before exploring further. # Try a Judge > Scores up to 5 cases' recorded replies with an unsaved Judge configuration. LLM Judges call their model (one call per case) and return the estimate and the actual cost; send `estimateOnly: true` to get the estimate first without calling. Nothing is saved, but an LLM Judge's model calls are billed to benchmark spend like a tiny run and count toward month-to-date spend (`costUsd` is what Zelto billed for the try, `listCostUsd` the provider list price); built-in and code Judges are free. ## OpenAPI ````yaml /docs-openapi.json post /v1/benchmarks/{id}/judges/try openapi: 3.1.0 info: title: Zelto REST API version: v1 description: >- Programmatic access to your Zelto organization. Every endpoint is scoped to the organization the API key belongs to. Mint a key at Settings → Integrations → API integration. servers: - url: https://api.zelto.ai description: Production security: - bearerAuth: [] tags: - name: Monitors description: Filtered, asynchronous exports of stored monitor results. - name: Agents description: Voice and text agents normalised through Zelto's ingestion pipeline. - name: Conversations description: >- Phone calls, chats, and voicemails — plus their transcripts and recordings. - name: Customer Metrics description: >- Customer-provided evaluations attached to a conversation (requires metrics enabled). - name: Reviews description: Human review state for conversations. - name: Buckets description: Reviewer routing — group conversations for batched review. - name: Companies description: Clients or brands your agents work for, and their logos. - name: Findings description: >- Structured issues discovered during analysis, with optional conversation attachments and comments. - name: Experiments description: >- Prospective measurement of two externally routed prompt versions across independent monitor outcomes. - name: Organizations description: >- Headless org provisioning. Requires an admin/service API key (not a read-only org-scoped customer key). - name: AI providers description: >- Your organization's own (bring-your-own) AI provider keys. Monitors can run on them instead of Zelto's built-in models. Owner or admin keys only; the secret is write-only. - name: Benchmarks description: >- Repeatable evaluations of agent prompts, models and speech-to-text engines over a fixed set of cases, scored by Judges. paths: /v1/benchmarks/{id}/judges/try: post: tags: - Benchmarks summary: Try a Judge description: >- Scores up to 5 cases' recorded replies with an unsaved Judge configuration. LLM Judges call their model (one call per case) and return the estimate and the actual cost; send `estimateOnly: true` to get the estimate first without calling. Nothing is saved, but an LLM Judge's model calls are billed to benchmark spend like a tiny run and count toward month-to-date spend (`costUsd` is what Zelto billed for the try, `listCostUsd` the provider list price); built-in and code Judges are free. operationId: tryBenchmarkJudge parameters: - name: id in: path required: true description: Benchmark id. schema: type: string format: uuid requestBody: required: true content: application/json: schema: type: object properties: judge: type: object properties: label: type: string minLength: 1 maxLength: 120 labelTemplate: type: string maxLength: 200 kind: type: string enum: - judge - code - builtin required: default: true type: boolean weight: default: 1 type: number minimum: 0.1 maximum: 10 checkGroupId: default: null anyOf: - type: string format: uuid - type: 'null' appliesTo: default: {} type: object properties: caseTypes: type: array items: type: string enum: - turn - conversation - audio tags: maxItems: 20 type: array items: type: string maxLength: 40 params: default: [] maxItems: 10 type: array items: type: object properties: key: type: string minLength: 1 maxLength: 40 label: type: string maxLength: 60 type: type: string enum: - number - text - percent default: anyOf: - type: number - type: string maxLength: 500 unit: type: string maxLength: 20 min: type: number max: type: number required: - key - label - type - default config: $ref: '#/components/schemas/BenchmarkJudgeConfig' toolCheck: type: string enum: - tool_call_match - tool_args_schema - tool_args_match required: - label - kind - config caseIds: minItems: 1 maxItems: 5 type: array items: type: string format: uuid estimateOnly: description: >- LLM Judges: return only the cost estimate, without calling the model. type: boolean required: - judge - caseIds responses: '200': description: Successful response. content: application/json: schema: $ref: '#/components/schemas/BenchmarkJudgeTryResult' '400': $ref: '#/components/responses/ValidationFailed' '401': $ref: '#/components/responses/Unauthorized' '403': description: >- `phi_mode_blocked`: The organization is in PHI mode, which allows only Bedrock-routed models and AWS Transcribe. content: application/json: schema: $ref: '#/components/schemas/BenchmarkError' '404': $ref: '#/components/responses/NotFound' components: schemas: BenchmarkJudgeConfig: oneOf: - type: object properties: kind: type: string const: judge rubric: type: string maxLength: 8000 model: type: object properties: provider: type: string enum: - platform - openai - anthropic - openrouter - vercel_gateway modelId: type: string minLength: 1 maxLength: 200 aiKeyId: type: - string - 'null' required: - provider - modelId scale: oneOf: - type: object properties: type: type: string const: pass_fail required: - type - type: object properties: type: type: string const: score_1_5 threshold: type: integer minimum: 1 maximum: 5 required: - type - threshold required: - kind - rubric - model - scale - type: object properties: kind: type: string const: code code: type: string maxLength: 20000 required: - kind - code - type: object properties: kind: type: string const: builtin rule: type: string enum: - reply_length - contains - regex_match - exact_match - wer_max - key_terms - tool_call_match - tool_args_schema - tool_args_match - transition_match - goal_reached - max_turns - path_includes required: - kind - rule description: >- Kind-specific Judge configuration: LLM rubric + model + scale, code, or a built-in rule. BenchmarkJudgeTryResult: type: object properties: results: type: array items: type: object properties: caseId: type: string passed: type: - boolean - 'null' value: type: string reasoning: type: - string - 'null' notApplicable: type: boolean error: type: - string - 'null' required: - caseId - passed - value - reasoning - error simulated: type: boolean description: >- True when LLM Judge verdicts were simulated because the Judge's model was unavailable. estimate: description: >- LLM Judges: the estimated model calls, tokens and list-price cost (USD) before calling. Null for built-in and code Judges. anyOf: - type: object properties: calls: type: integer inputTokens: type: integer outputTokens: type: integer costUsd: type: number model: type: string unpriced: type: boolean required: - calls - inputTokens - outputTokens - costUsd - model - unpriced - type: 'null' costUsd: description: >- LLM Judges: what Zelto billed for the try (USD), like a tiny run: list cost plus margin, or the bring-your-own-key fee. Counts toward month-to-date benchmark spend. Null for built-in and code Judges. type: - number - 'null' listCostUsd: description: 'LLM Judges: the provider list price (USD) of the calls made.' type: number tryId: description: >- LLM Judges: the try's billing id (one benchmark_spend charge per try). type: string required: - results - simulated description: >- Verdicts from trying a Judge on a few cases. Nothing is saved; an LLM Judge's model calls are billed to benchmark spend like a tiny run. BenchmarkError: type: object required: - error - code - message properties: error: type: string enum: - validation_failed - invalid_parameter - approval_required - plan_limit - phi_mode_blocked - owner_required - not_entitled - forbidden - not_found - payload_too_large - estimate_stale - pointer_moved - no_draft_changes - conflict - idempotency_key_reused - rate_limited - model_unavailable description: Stable error code (same value as `code`). code: type: string enum: - validation_failed - invalid_parameter - approval_required - plan_limit - phi_mode_blocked - owner_required - not_entitled - forbidden - not_found - payload_too_large - estimate_stale - pointer_moved - no_draft_changes - conflict - idempotency_key_reused - rate_limited - model_unavailable description: Stable, machine-readable cause; match on this. message: type: string description: A plain-English sentence saying what is wrong. hint: type: string description: The next step to try. details: type: object additionalProperties: true description: >- Code-specific context, for example `approvalRequestId` with `approval_required` or `cap` with `plan_limit`. Error: type: object required: - error properties: error: type: string description: >- Short error name: `Unauthorized`, `Validation failed`, `Not found`, or a stable snake_case code you can match on, such as `unknown_query_parameter`, `invalid_parameter`, `payload_too_large` or `not_found`. message: type: string description: A plain-English sentence saying what is wrong and how to fix it. code: type: string description: >- Stable, machine-readable cause; match on this rather than on `error` or `message`. On a 401: `missing_credentials`, `malformed_credentials`, `ingest_key_not_allowed` (a write-only observability ingest key was used on an endpoint that needs an API key), `key_expired` or `invalid_credentials`. Elsewhere, for example: `not_found`, `invalid_parameter`, `unknown_query_parameter`, `missing_parameter`, `validation_failed`, `invalid_json`, `payload_too_large`, `conversation_not_found`, `plan_required`. hint: type: string description: >- The next step to try: where to find the right id, which parameters the endpoint takes, how to upgrade the plan, how to authenticate. parameter: type: string description: 'With `invalid_parameter`: the query parameter that was rejected.' unknown: type: array items: type: string description: >- With `unknown_query_parameter`: the parameters that were sent but are not accepted here. valid: type: array items: type: string description: >- With `unknown_query_parameter`: the parameters this endpoint accepts. details: type: object additionalProperties: true responses: ValidationFailed: description: >- The request body or query string failed validation. `code` says which kind (`validation_failed`, `invalid_json`, `invalid_parameter`, `unknown_query_parameter`, `missing_parameter`), and an enum filter names the allowed values in `valid` (every repeated value is checked, e.g. `?status=open&status=zz` is a 400). A `validation_failed` body also carries `message` (the first failing fields), `hint` (how to fix it, for example that a finding `description` is a TipTap JSON document and not a string) and `details` (field name to messages). content: application/json: schema: $ref: '#/components/schemas/Error' Unauthorized: description: >- Missing or invalid API key. The body names the cause in `code` (for example `ingest_key_not_allowed` when a write-only ingest key is used here) with a `message` and `hint`. content: application/json: schema: $ref: '#/components/schemas/Error' NotFound: description: >- Resource not found in your organization. Also returned when a path id is not a UUID, and `{ "error": "not_found" }` for a path this API does not have. The body keeps `error: "Not found"` and adds `code: "not_found"` and a `hint`. content: application/json: schema: $ref: '#/components/schemas/Error' securitySchemes: bearerAuth: type: http scheme: bearer description: >- Send `Authorization: Bearer YOUR_API_KEY`. The key resolves to a single organization. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.