What a monitor is made of
A monitor is an ordered pipeline of stages, evaluated in sequence:
Filters run first and short-circuit: a call that fails any filter is dropped
before the AI stage runs, so it costs nothing. This is the cost control — put
cheap filters in front of expensive judgement whenever they can do the job.
Each
ai_analyze stage produces one typed output: a boolean, a number, an
option from a list, a string, a list of strings, or JSON. You designate one
boolean or number stage as the headline, and that value drives the affected
rate. A monitor with no headline is extraction-only: it still records values per
call, but trends volume rather than a rate.
A monitor with only filter stages is free to run — passing the filters is
itself the flag.
Creating a monitor
From Monitors → New monitor, describe what you want to watch for in a sentence and choose which agents it applies to. That opens the create workspace: an assistant on the left, and the monitor itself on the right. The assistant can pull real example calls, dry-run your filters against them, and propose a complete pipeline. Every proposal is applied live to the editor on the right, which you can also edit by hand at any point — the assistant builds on whatever is currently on screen.Try it before you save
The editor’s Try it step runs the rubric for real against calls you choose, before the monitor exists. Add up to 10 calls and press Run test; each one comes back with the monitor’s verdict, the value of every stage, and the model’s reasoning for it. This is the fastest way to tell a working rubric from a plausible-looking one. A common loop is: run the test, spot a call the monitor got wrong, tell the assistant which one and why, and re-run. The assistant sees your last test results, so “the second call is a false positive” refers to something it can actually act on. Three things to know:- Only calls with a transcript can be tested, so those are the only ones the picker offers. A monitor has nothing to judge without one.
- Test runs use the built-in models. If your monitor uses your own OpenRouter or Vercel AI Gateway key, save it first and it will run on your key from then on.
- Running a test is owner/admin-only, and so is simulating a call. Creating a monitor is open to every member — its ongoing per-call spend is the point. A test run is up to ten judge calls per click, which is discretionary one-off spend, so it is gated. A member sees the Try it step explaining that rather than a control they can’t submit.
Simulating a call
Sometimes you want to test a rubric for something that hasn’t happened yet — a new guardrail, a rare failure mode, a behaviour you’re trying to prevent rather than measure. Simulate a call generates a synthetic transcript against your agent’s live prompt from a one-line scenario (“a caller gets frustrated because the agent can’t find their booking”), and adds it to the test set alongside your real calls. Simulated calls are scratch material: they are judged like any other case, but they are never saved as conversations and never appear in your call history. For a fuller version of the same idea — reusable scenarios, pass/fail assertions, and prompt-vs-prompt comparison — see Simulations.Judging your recent calls
By default a monitor starts empty and only judges calls that arrive from the moment you save it. For a quiet agent that can mean waiting days to see whether it works. Also judge recent calls backfills instead: pick a window of up to 10 days and the monitor is applied to the calls already in it, so it has a trend immediately. The control shows how many calls fall in the window, and roughly what they cost, as you move the slider — because this is the one choice in the flow that spends model budget per call. It is off unless you turn it on, and it is owner/admin-only, like test runs. Members don’t see it. The backfill runs in the background after you save, judging the newest calls first. It stops at 5,000 calls, so an extremely high-volume org gets its most recent history rather than an unbounded bill. The slider says so before you save when your window is over the cap, and quotes the capped number rather than the one it would have to truncate.Setting your reading order
Monitors used to be listed newest-edited first, everywhere. That meant the order moved whenever anyone edited a monitor, so there was nothing to build muscle memory around when reviewing a call. Drag a monitor by its handle on Monitors — in either the Graph or the List view — to set your own reading order. The order applies wherever monitors are listed for you, including the Monitors panel inside a conversation, which is where it matters most: the panel reads top to bottom while you work a call. You can also reorder from the keyboard: tab to a monitor’s handle, press space to pick it up, move it with the arrow keys, then press space to drop it (escape cancels). Three things worth knowing:- The order is yours alone. Rearranging it changes nothing for your teammates, so you can order monitors around what you personally review first.
- A new monitor never disturbs it. Monitors you have not placed sort after the ones you have, so creating a monitor adds it to the end rather than reshuffling an order you set. Reset order on the Monitors page clears your arrangement and returns you to the default.
- Reordering while filtered only moves what you can see. If you have narrowed the page to one agent, dragging rearranges those monitors among the positions they already hold. Everything the filter is hiding keeps its place.
Reading a monitor
A monitor’s detail page has four tabs:- Overview — the affected-rate trend over a rolling window.
- Results — every judged call, with its verdict and per-stage values.
- Configuration — the pipeline, agents, and model. Editing bumps the monitor’s version; past results keep the version they were judged under, and are not re-judged.
- Alerts — see below.
Alerts
On the Alerts tab, set a threshold and Zelto notifies you when the monitor crosses it: a comparator (above or below), a percentage, a rolling window, and optionally a number of consecutive days the breach must persist before it fires. Delivery goes to a Slack channel, a list of email addresses, or both. Alerts are evaluated on a daily sweep, so expect a first notification within a day of the breach beginning, plus any sustained-days requirement you set.From a finding to a monitor
A finding is a curated issue with a set of linked calls; a monitor is a continuous check. When a finding turns out to describe something recurring, promote it: from the finding’s actions menu, Promote to monitor creates a monitor seeded with the finding’s existing calls, so the trend starts with the history you already have.Over MCP and the agent
Monitors are fully manageable from the MCP server:create_monitor
(with either a plain-language rubric or a starterKey from
list_starter_monitors), list_monitors, set_monitor_enabled,
delete_monitor, promote_finding_to_monitor, and save_monitor_alert.
Note that update_monitor only renames, pauses, or relabels. To change a
monitor’s rubric or its agent set, edit it in the app, or delete it and create a
new one.
Related
- Agents — what a monitor is assigned to.
- Conversations — where a call’s monitor results appear.
- Findings — curated issues, and where promotion starts.
- Simulations — reusable synthetic scenarios with assertions.
- Slack — where alerts land.