> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zelto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Compare real-world calls from two prompt-version groups using the monitors you choose.

An **experiment** compares two groups of real calls to help you decide whether a
prompt change is an improvement. Each group, called an **arm**, is made up of
specific deployed prompt versions. Attach the [monitors](/docs/monitors) that
matter to the change and use their results to compare the arms.

Experiments look at traffic your agents have already handled. They do not route
callers, change an agent's live prompt, or create synthetic calls. To test a
candidate prompt before it is deployed, use [Simulations](/docs/simulations)
instead.

## What an experiment contains

| Part                | What it means                                                                                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Arm A and Arm B** | The two non-overlapping sets of deployed prompt versions to compare. Give them labels such as **Control** and **Warmer greeting** to make the result easier to read. |
| **Prompt versions** | The exact versions whose calls belong to an arm. An arm can include versions from more than one [agent](/docs/agents), with up to 20 versions per arm.               |
| **Monitors**        | The checks whose results you want to compare. They are optional, and attaching one does not change how it runs on future calls.                                      |
| **Window**          | The calls made from the experiment's start time until it is concluded. You can back-date the start time to include calls already made by the selected versions.      |

<Note>
  The same prompt version cannot be in both arms. If it were, the same call could
  be counted on both sides, which would make the comparison unreliable.
</Note>

Only deployed prompt versions can be selected. A draft has not handled calls,
so it cannot contribute real-world evidence yet.

## Create an experiment

1. Open **Experiments** in the sidebar and select **New experiment**.
2. Give the experiment a clear name. Add a description that records the change
   you are testing and the outcome that would make you adopt it.
3. Optionally choose a **Start of window**. Leave it blank to begin collecting
   calls now, or back-date it to include earlier calls from the versions you
   selected.
4. Build **Arm A**. Select an agent, then select one or more deployed prompt
   versions. You can choose another agent and add its versions to the same arm.
5. Build **Arm B** the same way. A version already in Arm A is unavailable in
   Arm B, and vice versa.
6. Select the monitors you want to compare, then select **Create experiment**.

The experiment opens to its setup page. It lists each arm's agents and prompt
versions, the selected monitors, and the start time of the call window.

## Choose useful arms

An arm is a population of calls, not a label for an idea. Keep the comparison
focused on the prompt change you want to evaluate.

* Use **Control** for the deployed version that represents current behavior and
  a descriptive label for the alternative, such as **Shorter opener**.
* Keep each version in one arm only.
* Include only versions whose calls are relevant to the question. Combining
  unrelated prompts or agents can obscure the effect you are trying to measure.
* Pick monitors that express the desired outcome. For example, a warmer greeting
  may be compared on a monitor for caller frustration and one for successful
  appointment booking.

<Tip>
  Use a narrow, intentional start time when you want to compare a recent change.
  Use a back-dated window when the selected versions already have enough relevant
  traffic to answer the question.
</Tip>

## Read the results

The results table has one row per attached monitor. Each arm shows its headline
number and, beneath it, how many calls that number is based on.

| Column            | What it means                                                                                                                                                       |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Arm A / Arm B** | The monitor's affected rate, or its average score, for that arm — followed by the number of judged calls and the arm's coverage.                                    |
| **Verdict**       | Which arm is ahead, the size of the gap, and the p-value. For a monitor that flags a problem, the **lower** rate wins; for a scored monitor, the higher score wins. |

A verdict appears only once both arms have at least 20 judged calls and the gap
is statistically significant. Until then the verdict reads **Needs 20 judged
calls on each side**; a gap that could plausibly be noise reads **No clear
difference** rather than naming a winner.

<Note>
  Rates here divide by **judged** calls, not by every call. A monitor's own page
  divides by every call its agents handled, so the two numbers can differ. A call
  with no verdict is unknown, not unaffected — counting it as unaffected would
  understate every rate and make it drift as more calls are judged.
</Note>

## Coverage and evaluating missing calls

A call only carries a monitor's verdict if that monitor was enabled when the
call was processed. An experiment that reaches back before a monitor existed
therefore starts at low **coverage** — the share of eligible calls that have a
verdict — and its comparison rests on whatever sample happens to be judged.

The banner above the results table shows the weakest arm's coverage and how many
calls are still unjudged. Select **Evaluate missing calls** to judge them.

<Warning>
  Evaluating spends AI credits. Each unjudged call is run through every attached
  monitor it is missing, so the cost scales with both the number of calls and the
  number of monitors. The banner states the number of calls and judge runs before
  you confirm, and a single run is capped at 5,000 calls.
</Warning>

Evaluation runs in the background, newest calls first, and results appear as
they land. It only judges calls that are missing a verdict, so running it twice
costs nothing the second time. It stays available after an experiment is
concluded — concluding fixes which calls are compared, not how many of them have
been judged.

## Run, conclude, or reopen

A new experiment is **Running**. Calls from its selected versions that occur
after the start of the window belong to the appropriate arm as they arrive.

Select **Conclude** when you no longer want new calls to enter the comparison.
This sets the end of the window but keeps the experiment and its selected
monitors intact. You can still work with the calls already inside the window.
Select **Reopen** if you want the experiment to begin including new calls again.

Deleting an experiment removes the experiment and its arm assignments. It does
not delete the agents, prompt versions, calls, or monitors it referenced.

## Experiments and simulations

Use both tools at different points in a prompt-change workflow:

1. Use [Simulations](/docs/simulations) before deployment to test a candidate
   prompt against controlled, synthetic caller scenarios.
2. Deploy the prompt version once it is ready for real traffic.
3. Create an experiment to compare real calls from that version with a control
   version on the monitors that reflect your goal.

Simulations answer, “How does this prompt behave under the cases I designed?”
Experiments answer, “How did these deployed prompt versions behave in actual
calls?”

## Related

* [Agents](/docs/agents) — where prompt versions and their call history live.
* [Monitors](/docs/monitors) — the continuous checks you attach to an experiment.
* [Simulations](/docs/simulations) — test a candidate prompt before it reaches
  real callers.