Monitors are included in the Observability and Improve plans. On the Free plan the Monitors page shows what monitors do and which plan unlocks them. See Plans, limits and retention.
Monitor selected companies
Choose Companies, search for the companies you want, and select at least one. Matching uses the company attributed to each conversation. All agents handling conversations for those companies are covered automatically, including future agents. Conversations without a company, and conversations attributed to an unselected company, are excluded. Agent membership changes do not rewrite historical coverage. You can save a monitor for companies that have no agents or conversations yet. It shows No matching conversations yet until matching results arrive. This applies to every monitor kind, including audio monitors. Change the selection in the monitor’s configuration. History, rates, exports, and alerts use the current selection; stored results remain intact. Sidebar company filters and conversation metadata filters narrow that selection further. Historical conversations are evaluated only when you explicitly choose the existing historical analysis option. A queued backfill cannot add companies beyond the selection you authorized. Deleting the last selected company leaves a monitor that matches no conversations. Disabling the Companies feature does not remove or broaden saved scopes. Model comparisons from an older scope are marked stale; run a new comparison to evaluate the current population.What a monitor is made of
A monitor is an ordered pipeline of stages, evaluated in sequence:
Filters run first and short-circuit: a conversation that fails any filter is dropped
before later stages run, so it costs nothing. This is the cost control — put
cheap filters in front of expensive judgement whenever they can do the job.
Each
ai_analyze stage produces one typed output: a boolean, a number, an
option from a list, a string, a list of strings, or JSON. You designate one
boolean, number, flow, tool-check, or code stage as the headline, and that
value drives the affected rate. A monitor with no headline is extraction-only:
it still records values per conversation, but trends volume rather than a rate.
For each AI step, choose Include agent prompt and Include conversation context
(CRM) independently in the editor. Both are off on new steps; existing steps
keep their settings (older steps without explicit settings include the agent
prompt and exclude CRM context). Steps with the same settings share a model
request when no code step separates them. Different settings or an intervening
code step use separate requests, reflected in cost estimates.
Flow checks always include the agent prompt and exclude CRM context. A step
does not receive another context group’s AI results. Real-conversation previews include
available CRM data only for steps that enable it.
A monitor with only filter stages is free to run — passing the filters is
itself the flag. Positive, non-negated filter_regex stages also record the
matched text, transcript fragment, role, and turn index automatically, without
code or an LLM. This evidence is included in
monitor.matched webhooks, with
documented size limits and handling of matches across turns.
A monitor with code stages and no AI stages costs **$1 per 100,000 successful
results** ($0.00001 per result), once per monitor per conversation, regardless
of how many code stages it has. Failed, unscored, and filtered-out evaluations
are not charged. Mixed AI/code monitors keep their existing AI pricing without
an extra code-result charge. Result retries remain free by default.
Tool checks
Use a tool check when the question is whether the agent actually did something — calledsubmit_order, booked the slot, looked the customer up —
not whether it said it did.
The stage matches expected tool names on tool turns (case and punctuation
insensitive: submit_order, submitOrder, and submit-order are the same
tool) and classifies the conversation from those events:
Two rules that matter in production:
- Confirmation language is never success. An optional claim-pattern regex
over agent turns records that the agent said the action completed. That is
a claim, not evidence the tool ran. The order-placed case — “your order has
been placed” with no
submit_orderevent — is flagged as claimed without a successful call. - Missing tool metadata is not a pass. Conversations ingested before tool events were captured, or from a provider that does not emit them, get insufficient evidence (no headline score). They are not treated as successful execution and they are not treated as a proven miss.
Code stages
Use a code stage when the rule is deterministic and specific enough that you would rather write it than describe it to a model. You write a small TypeScript snippet (no imports) that runs once per conversation and returns a value. Code-only monitors use the per-result pricing above and make no model calls. Code runs the same way in the “try it” preview, in live judging, and when backfilling. The snippet receives one argument,input:
Using an earlier AI result
Place an AI analysis step before the code step. Under the code editor, Earlier step results lists the available steps and their output types. Click Copy reference beside a step to copy an expression that reads its value by its stable step ID. For this example, use a Number AI output with bounds that cover the amounts you expect (for example, 0–1000), and a Yes / no code output. Numeric results are clamped to the AI step’s configured bounds. Paste the reference into your code:stageId, label, kind, value, reasoning, and,
when applicable, missingReferenceFields. Earlier code, flow-check, and
tool-check results are available too; filters do not produce results.
The first step receives an empty array. Only steps before the current code
step are included. AI steps after the code have not been requested yet.
Renaming a step preserves its reference. Deleting it or moving it after the
code step makes its result absent (undefined in the example); a step that
could not produce a value has value: null. Check the value before using it.
A thrown error records a null code result with the error explanation, and the
pipeline continues. Changing input.results inside the sandbox does not change
the stored results or what later steps see.
This works in previews, live evaluation, and backfills, within the same monitor.
Results are scoped to that monitor. Existing saved results are unchanged until
re-evaluated.
Two extra moves let a code stage shape the rest of the pipeline:
exit(value)short-circuits. A plainreturnrecords the value and lets the pipeline continue.exit(value)records the value and stops — every later stage is skipped, including a following AI stage and its cost. Put a cheap code check in front of an expensive AI stage andexitearly when the answer is already known.- Local results before the first AI stage flow into every model group. A
code stage that
returns before the first AI stage contributes its value as a ground-truth signal. Later code results and other groups’ AI results are not added to those shared signals.
input — an infinite loop
or a runaway allocation is stopped, not the platform. Imports are rejected, and
the snippet’s syntax is checked when you save the monitor.
Run an evaluation only above a frustration threshold
Arrange three steps in the same monitor:- AI analysis — Frustration: return a Number from 0 to 100 measuring the customer’s frustration.
-
Code — Frustration threshold: choose a Yes / no output and copy the
Frustration step’s reference from Earlier step results into this code:
- AI analysis — Complaint handling: evaluate whether the agent handled the customer’s complaint appropriately. This step runs only when the frustration score is greater than 70.
exit(false)
records the code result and skips every remaining step. No request is sent or
AI tokens charged for the skipped evaluation. The initial frustration analysis
still runs and incurs its normal cost. A plain return false would continue
into the next evaluation; use exit(false) to stop.
The skipped evaluation has no result. If it is the monitor’s headline, its score
is unavailable; the code’s exit value does not replace the configured headline.
Every code step separates model batches so its decision takes effect before
later AI requests. When the code continues, that separation can mean more model
requests than a pipeline without the code. Cost estimates assume all code gates
continue; actual usage includes only requests that run.
Creating a monitor
From Monitors → New monitor, search or browse the built-in templates. Every organization gets the same catalog. Templates show the values they produce before you choose one, and selecting a template opens its editable prompt so you can tailor it before continuing. Choose Create custom monitor when you want to start from a plain-language description instead. Describe what you want to watch for in a sentence and choose which agents it applies to. That opens the create workspace: an assistant on the left, and the monitor itself on the right. The assistant can pull real example conversations, dry-run your filters against them, and propose a complete pipeline. Every proposal is applied live to the editor on the right, which you can also edit by hand at any point — the assistant builds on whatever is currently on screen.Audio evaluations
The Audio evaluations template category contains monitors that score the recording itself rather than its transcript. Use them to track audio quality, response latency, and pronunciation or language accuracy. After choosing a template, set its threshold and agent scope, then create it like any other monitor.Try it before enabling
The editor’s Try it section runs the current configuration against up to 10 conversations you choose. It is available when creating a monitor and in a saved monitor’s Configuration tab, including while paused. Press Run test to see the verdict, each stage’s value, reasoning, and any evaluation errors. Tests include unsaved changes and do not enable the monitor or save results to the conversations. AI steps consume credits or your provider’s tokens. After changing the configuration, run the test again: earlier results are marked out of date. A completed test confirms execution; review the verdicts to assess accuracy. This is the fastest way to tell a working rubric from a plausible-looking one. A common loop is: run the test, spot a conversation the monitor got wrong, tell the assistant which one and why, and re-run. The assistant sees your last test results, so “the second conversation is a false positive” refers to something it can actually act on. Three things to know:- Only conversations with a transcript can be tested, so those are the only ones the picker offers. A monitor has nothing to judge without one.
- Test runs use the selected provider and model, including your OpenRouter, Vercel AI Gateway, or TypeSafe key. Saved monitors can reuse their saved key, and a key stored once in Settings → AI providers can be picked by any monitor.
- Running a test is owner/admin-only, and so is simulating a conversation. Creating a monitor is open to every member — its ongoing per-conversation spend is the point. A test run evaluates up to ten conversations per click, with a model request for each context group reached per conversation, so it is gated. A member sees the Try it step explaining that rather than a control they can’t submit.
Simulating a conversation
Sometimes you want to test a rubric for something that hasn’t happened yet — a new guardrail, a rare failure mode, a behaviour you’re trying to prevent rather than measure. Simulate a conversation generates a synthetic transcript against your agent’s live prompt from a one-line scenario (“a user gets frustrated because the agent can’t find their booking”), and adds it to the test set alongside your real conversations. Simulated conversations are scratch material: they are judged like any other case, but they are never saved as conversations and never appear in your conversation history. For a fuller version of the same idea — reusable scenarios, pass/fail assertions, and prompt-vs-prompt comparison — see Simulations.Judging your recent conversations
By default a monitor starts empty and only judges conversations that arrive from the moment you save it. For a quiet agent that can mean waiting days to see whether it works. Also judge recent conversations backfills instead: pick a window of up to 10 days and the monitor is applied to the conversations already in it, so it has a trend immediately. The control shows how many conversations fall in the window, and roughly what they cost, as you move the slider — because this is the one choice in the flow that spends model budget per conversation. It is off unless you turn it on, and it is owner/admin-only, like test runs. Members don’t see it. The backfill runs in the background after you save, judging the newest conversations first. It stops at 5,000 conversations, so an extremely high-volume org gets its most recent history rather than an unbounded bill. The slider says so before you save when your window is over the cap, and quotes the capped number rather than the one it would have to truncate.Setting your reading order
Monitors used to be listed newest-edited first, everywhere. That meant the order moved whenever anyone edited a monitor, so there was nothing to build muscle memory around when reviewing a conversation. Drag a monitor by its handle on Monitors — in either the Graph or the List view — to set your own reading order. The order applies wherever monitors are listed for you, including the Monitors panel inside a conversation, which is where it matters most: the panel reads top to bottom while you work a conversation. You can also reorder from the keyboard: tab to a monitor’s handle, press space to pick it up, move it with the arrow keys, then press space to drop it (escape cancels). Three things worth knowing:- The order is yours alone. Rearranging it changes nothing for your teammates, so you can order monitors around what you personally review first.
- A new monitor never disturbs it. Monitors you have not placed sort after the ones you have, so creating a monitor adds it to the end rather than reshuffling an order you set. Reset order on the Monitors page clears your arrangement and returns you to the default.
- Reordering while filtered only moves what you can see. If you have narrowed the page to one agent, dragging rearranges those monitors among the positions they already hold. Everything the filter is hiding keeps its place.
Reading a monitor
On the main Monitors page, use 1d, 7d, 15d, or 30d for a rolling view through today. Choose Custom to select one UTC calendar day or an exact start and end date; future dates are unavailable. The range filters the headline values, Graph, List, and Compare views, and is stored in the URL so you can bookmark or share it. Overview cards and rows show Active/Paused status and scored coverage for the current monitor version. Historical trend charts use all versions and their labeled transcript-bearing-conversation denominator. Open Monitors → Reports to browse saved reports and download PDFs. Owners and admins can select monitors in Overview and choose Create report, or use Add to report on a monitor. New drafts retain company, metadata and dates; agent/group filters are not saved and are disclosed before saving. Reports accept up to 20 monitors and 31 calendar days. Open a report, choose the previous week, this week or explicit dates, then use Download PDF in its header. CSV/JSON, evaluation details and delivery history are expandable. Weekly delivery is opt-in. When exports are enabled, all members can use Create export from a monitor, selected monitors, or Monitors → Exports. Choose CSV, Excel or Parquet for up to 20 monitors and 100,000 conversations, set the export’s own filters and period in the displayed timezone, then check the matching-conversation count before creating it. Historical result versions are included by default; select Only current version to restrict the export. Each row contains one conversation’s stored results and stage values, with the actual evaluated version for each monitor. Replaced evaluations are not a revision log. Reasoning is optional, and transcripts and conversation metadata are excluded. The organization shares the history and can download completed files for 30 days or recreate failed or expired exports with current data. Download table remains the immediate download for the results table. Before downloading all matching results, Download table checks the count: direct CSV downloads support up to 10,000 rows, and review downloads up to 1,000 cases. If the selection exceeds the limit, choose Current page, narrow the table filters, or open Exports for a larger export. Exports use their own period and filters; table filters are not copied automatically. Use Today, Last 7 days or Last 30 days to set the export period quickly. Today starts at midnight in the displayed timezone; the other shortcuts use rolling windows. All end at the instant you select them. Editing either bound clears the shortcut selection without changing other filters. In the history, Reuse filters opens a new request from an existing queued, generating or ready export. The saved period, monitors, conditions and other settings are filled in for review and editing. Keep the same file format or choose another; each request uses current data and leaves the original file available. Failed and expired exports offer the same flow as Create again. In the history, Change format is available for queued, generating and ready exports. Choose a different CSV, Excel or Parquet format to create another export with the saved filters and current data. The original request and file stay unchanged; the new export keeps the saved version selection. Current-only exports resolve monitor versions when generation starts. To notify an external receiver when the file is ready, choose an active approved destination under Notify when complete, or leave No notification selected. Owners/admins approve destinations in Webhooks. Delivery status is separate from file generation.Export with an API key
Your backend can request filtered exports with an organization API key; a browser session is not required. Monitor exports must be enabled and the organization must have Monitors access. UseAuthorization: Bearer YOUR_API_KEY; do not send a slug
or organization ID. Ingest-only keys cannot read exports.
POST https://api.zelto.ai/v1/monitor-exportswith a UUIDoperationIdandconfig. Replace the example monitor ID and generate your own operation ID.- Poll
GET https://api.zelto.ai/v1/monitor-exports/{id}with the same key, every five seconds, untilstatusiscompleted,failedorexpired. - For a completed export, call
GET https://api.zelto.ai/v1/monitor-exports/{id}/downloadwith the key. Follow the 307 redirect to download the file without forwarding your API key to storage. The signed URL lasts at most 15 minutes and must be kept private.
operationId, configuration and key owner when retrying a request
after a timeout; this recovers the original export instead of creating duplicate
work. Reusing it with a different configuration returns 409. A new export requires
a new operation ID. Requests return 202, including replays; inspect the returned
status rather than assuming it is still pending.
The configuration supports companyId, agentId, groupId, up to eight
metadata conditions and per-monitor result (all, affected, not_affected).
match: "all" or "any" combines explicit result conditions; metadata conditions
always combine with AND. since is inclusive and until exclusive, using call time
(startedAt, falling back to ingestion time). versionScope defaults to
historical; current resolves versions when generation starts. Limits, formats,
retention and optional reasoning are the same as the platform export above.
These endpoints export stored results, including automatically captured regex
evidence when available; they do not reprocess conversations or extract additional
keywords. Historical results and filters without a positive regex match can have
empty stage values.
When export webhooks are enabled, notificationEndpointId can identify an active,
admin-approved destination in your organization; the key owner must still be a
member. This sends monitor_export.completed when ready. See
Webhooks for the payload
and signature verification. Never put an API key in a URL or a webhook payload.
A missing export returns 404, a file still generating returns 409 on download,
and an expired file returns 410. Disabled exports or temporary failures return
503. Every request checks current key access; downloads are audited.
Filtering by conversation metadata
The Metadata bar on the Monitors page narrows the conversations every number is computed over, not which monitors are listed. Add a filter row, pick a property key (dotted paths into the conversation’s provider metadata, such ascampaign or
model.provider; keys seen on your recent conversations are suggested), choose an
operator (equals, does not equal, contains, does not contain, numeric
greater than / less than and their or equal forms, is set, is not set, is true, is false), and enter a value. Up to eight rows combine with
AND. These are the same property filters as the
Conversations list.
With a filter active, every affected rate, conversation count, and chart on the list,
every column on Compare, and a monitor’s Overview trend, stage graphs,
flow funnel, and Results rows count only the conversations whose metadata matches.
The filter lives in the URL as ?meta=key:op:value, so a filtered view is
shareable, it stays applied as you switch between Graph and List, change the
time range, or move between a monitor’s tabs, and it follows you from the list
into a monitor and onto Compare. On Compare the bar appears once two or more
columns are selected.
Opening a monitor from Overview or a report keeps the selected dates and filters
in a reading view with current-version coverage and conversations. Report drill-down also
retains its saved timezone. Use the return link to resume the same period, or
Open monitor configuration to leave that scope and access the full detail page:
- Overview — the affected-rate trend plus graphs for each chartable pipeline stage.
- Results — every judged conversation, with its verdict and per-stage values.
- Benchmark — compare the Fast, Balanced, and Advanced model tiers on an automatic sample of conversations this monitor already judged.
- Configuration — the pipeline, agents, and model. Editing bumps the monitor’s version; past results keep the version they were judged under, and are not re-judged unless you request a retry or enable automatic retries for errored results.
- Alerts — see below.
Retry errored results
In the monitor’s Results tab, owners and admins can select individual errored results, or all eligible errors on the current page, then click Retry selected. Each selected result is queued for one immediate retry using the current monitor settings. This works with automatic retries turned off. Rows with an active retry show Retry queued and cannot be selected again until the attempt finishes. Selections apply to the current page (up to 50 results); changing the page or filters clears the selection. Paused monitors, audio monitors, and company-scoped views do not offer manual retries. Monitor-result retries are free by default; a platform administrator can enable billing for future retries. In Configuration → Retries, owners and admins can enable Automatically retry errored results and choose a delay from 1 minute to 7 days. Retries are off by default. Each errored conversation gets at most one automatic retry per monitor version, using the current monitor settings. This includes existing errors; their delay starts when you save the setting. New errors wait from their evaluation time. The worker checks for due retries about once a minute, so a busy queue can delay the actual start. The same retry billing policy applies to automatic retries. A completed retry replaces the stored result and updates reporting. Successful results, conversations excluded by filters, and missing-reference results are not retried. Recording-based audio monitors do not use this setting. Turn the toggle off or pause the monitor to cancel automatic work that has not started. Turning off automatic retries does not cancel explicitly selected retries. If the retry also errors, it remains an error rather than retrying indefinitely. Owners and admins can benchmark up to 1,000 transcript-bearing conversations already judged under the monitor’s current version. Zelto builds the dataset automatically, balancing affected and unaffected stored verdicts when volume allows, and runs the comparison in background batches. Each tier reports precision, containment, observed price, and projected price per 1,000 conversations; the best-fit marker ranks precision, then containment, then cost. The stored verdict is the provisional label. After the run, the Benchmark tab shows only conversations where the stored verdict or model tiers disagree. Open a conversation to inspect its transcript, then select the correct outcome; precision updates immediately from that human correction. Conversations stopped by deterministic filters are excluded from containment because the AI judge never sees them. Benchmark runs never change the monitor or its production results. The stage graphs cover the most recent 30 days. Boolean and enum stages show how their results are distributed over time, numeric stages show the daily average, and tool checks show their outcome mix. Flow checks keep their step-completion funnel and drop-off summary. Free-text, string-list, and JSON extraction outputs stay on individual results because grouping them would hide their meaning. A conversation shows N/A for a monitor that is assigned to its agent but has not judged it — usually because the conversation predates the monitor and was not backfilled, or because a filter dropped it. In the Results table, select Inspect evaluation on a judged conversation to open its evaluation trace. The interactive graph follows the transcript through each filter and evaluation stage to the final outcome. Select a node to see the evidence behind it: regex filters show the exact scoped text, match, and transcript turn; AI stages show their instruction, conversation-time input, saved output, reasoning, and model. If the monitor was edited after the conversation was judged, the trace warns that its graph uses the current pipeline while the displayed verdict remains the saved historical result.Alerts
On the Alerts tab, set a threshold and Zelto notifies you when the monitor crosses it: a comparator (above or below), a percentage, a rolling window, and optionally a number of consecutive days the breach must persist before it fires. Delivery goes to a Slack channel, a list of email addresses, or both. Alerts are evaluated on a daily sweep, so expect a first notification within a day of the breach beginning, plus any sustained-days requirement you set.From a finding to a monitor
A finding is a curated issue with a set of linked conversations; a monitor is a continuous check. When a finding turns out to describe something recurring, promote it: from the finding’s actions menu, Promote to monitor creates a monitor seeded with the finding’s existing conversations, so the trend starts with the history you already have.Over MCP and the agent
Monitors are fully manageable from the MCP server:create_monitor
(with either a plain-language rubric or a starterKey from
list_starter_monitors), create_pipeline_monitor, list_monitors,
get_monitor_results, update_monitor, set_monitor_enabled,
delete_monitor, promote_finding_to_monitor, and save_monitor_alert.
update_monitor renames, pauses, or relabels a monitor, replaces the agents it
scores (agentIds; pass [] for every agent in the organization, audio
monitors included), or replaces its company selection (companyIds, at least
one). agentIds and companyIds are mutually exclusive; omitting both preserves
the saved scope. Creation tools also accept companyIds, and list_monitors
returns the scope and selected companies. Company scope edits update visible
history without deleting stored results or automatically evaluating old conversations.
The tool also changes the sample rate (sampleRate, 1–100). To change a
monitor’s rubric or pipeline without losing its results, use update_monitor_pipeline
(owner or admin only; see MCP). Before authoring a tool_check,
call list_agent_tools for the exact tool names agents have invoked.
To change an existing AI step’s context, read its stage ID with get_monitor,
then pass stageContext: [{ stageId: "step-id", useAgentPrompt: false, useCallContext: true }] to update_monitor. Omitted options stay unchanged;
all supplied edits apply atomically. Context changes increment the monitor
version for future evaluations; historical results remain unchanged.
Related
- Agents — what a monitor is assigned to.
- Conversations — where a conversation’s monitor results appear.
- Findings — curated issues, and where promotion starts.
- Simulations — reusable synthetic scenarios with assertions.
- Slack — where alerts land.
- AI providers — run monitors on your organization’s own OpenAI, Anthropic, OpenRouter, Vercel AI Gateway, or TypeSafe key.
TypeSafe models with your own key
Select TypeSafe (BYOK) in the monitor’s model settings, choosejev-latest
(or enter a TypeSafe model ID), and supply your TypeSafe API key. Zelto encrypts
the key per monitor. Leaving the key blank when editing keeps the saved key only
when the provider is unchanged. Switching providers requires a new key.
TypeSafe supports these AI check outputs:
- Yes / no: a Noul probability of at least 50% becomes
true. - Category: Choice returns one of your unique category options.
- Number: enter 2–10 descriptive rubric levels, one per line, from lowest to highest. The fractional level score maps evenly onto your minimum and maximum (defaults: 0–100). A score of 1.5 across three levels maps to 7.5 on a 0–10 range. Use these scores for judgments, not numeric extraction.
Evaluate recorded trace evidence
Enable Use the call trace on an AI or code check to evaluate the call’s recorded steps, inputs, outputs, errors, timings, and logs alongside its transcript. AI checks can cite span IDs and log timestamps. Code checks readinput.trace.spans and
input.trace.logs, for example:
Send monitor results to your system
Use outbound webhooks to send each new positive monitor result to your database or CRM. Configure destinations under Settings → Integrations → Webhooks, verify a test, and deduplicate received events by their event ID. These per-conversationmonitor.matched events are separate from weekly aggregate report webhooks.
