> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zelto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Start sending traces to Zelto

> Create an ingest key, export your first trace, identify agents with your own IDs, and connect traces to calls.

Send a test trace, find it in Zelto, then apply the same setup to your agent.
A **trace** represents one call or session; its **spans** represent timed steps
such as speech recognition, an LLM request, a tool call, or speech synthesis.
Exporting traces does not upload a call transcript or recording: those arrive
through your [call integration](/docs/integrations).

You need a Zelto organization, an `owner` or `admin` to create its ingest key,
and a server or worker that can make HTTPS requests to `ingest.zelto.ai`.
The first example uses Python 3.10+ and synthetic data; it does not place a call.

## 1. Create a telemetry key

In the organization you want to send to, open **Traces → Connect agent** or
**Settings → Integrations → Observability**. Create an **observability ingest
key** and copy it when shown. It is a write-only credential for that organization.
Store it in your worker's secrets or environment as `ZELTO_INGEST_KEY`.

This is separate from the REST/MCP API key used to upload conversations.
Telemetry uses `Authorization: Bearer <ingest-key>` and needs no
`X-Zelto-Provider` header. Keep the key in server-side code.

## 2. Export a first trace

Create an isolated Python environment and install the SDK and HTTP exporter:

```bash theme={null}
python3 -m venv .venv
source .venv/bin/activate
python -m pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
export ZELTO_INGEST_KEY="<your-observability-ingest-key>"
```

Save this as `send_trace.py`, replacing the placeholder key in your environment
before running it. This example represents one agent, so its stable identity is
on the **resource**, the attributes shared by all spans from this provider:

```python theme={null}
import os
import uuid

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({
    "service.name": "zelto-trace-demo",
    "zelto.agent_external_id": "support-agent",
    "zelto.agent_provider": "zelto",
}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://ingest.zelto.ai/v1/traces",
    headers={"Authorization": f"Bearer {os.environ['ZELTO_INGEST_KEY']}"},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("zelto-getting-started")
call_id = f"demo-{uuid.uuid4().hex}"

try:
    with tracer.start_as_current_span("demo.call", attributes={"session.id": call_id}) as root:
        with tracer.start_as_current_span("demo.tool", attributes={
            "session.id": call_id,
            "tool.name": "check_availability",
        }):
            pass
        print(f"trace_id={root.get_span_context().trace_id:032x}")
        print(f"session.id={call_id}")
    provider.force_flush(timeout_millis=10_000)
finally:
    provider.shutdown()
```

```bash theme={null}
python send_trace.py
```

You should see a trace ID and a unique session ID. The child span shares the
root's trace ID because it is created inside the active root span. Attributes
such as `session.id` are **not automatically inherited** by child spans, so
set them on every relevant span or through your framework's metadata mechanism.
The script flushes ended spans before exiting; inspect exporter errors too—a
printed ID or a completed flush alone does not prove the server accepted it.

## 3. Find it in Zelto

Open **Traces** in the same organization, clear agent/company filters, and
choose a range that includes now. Look for `demo.call` and compare its trace ID
with the script's output. Open it to see `demo.tool` beneath the root and inspect
its attributes. Zelto reuses or creates the agent named `support-agent`.

A successful HTTP response acknowledges acceptance. For organizations using
buffered ingestion, data becomes queryable asynchronously; the target is
within **30 seconds after acceptance under normal load**, not a guaranteed
maximum. Exporter batching adds time before acceptance, and processing can take
longer during an outage. Refresh before diagnosing a missing trace. Linked-call
panels also require the conversation to have arrived.

This synthetic example has **no linked conversation** until a call with the same
external ID is uploaded. That does not mean the trace export failed.

## Identify agents and calls consistently

| Identifier | Use | Lifetime |
| - | - | - |
| `zelto.agent_external_id` | The ID of the agent in your system; use the same value as `agent.externalId` in call uploads. | Stable across calls and releases. |
| `zelto.agent_provider` | The same provider namespace as call ingestion, for example `livekit`. | Stable for that integration. |
| `session.id` | The call, session, or room ID; match `call.externalId` in call uploads. | Unique per call; unchanged on retries. |
| Trace ID / span ID | OpenTelemetry identifiers generated by your SDK. | One trace per session, one span ID per operation; unchanged on retries. |

Zelto matches external agent IDs **within your organization**. Unknown IDs create
agents automatically, initially named after the external ID. If you omit the
provider, Zelto reuses a unique existing match or creates a new agent under
`zelto`. Send `livekit` when calls use `X-Zelto-Provider: livekit` to prevent
separate identities when telemetry arrives first. Multiple matches across
providers remain unassigned; specify the provider to disambiguate them.

External IDs are case-sensitive strings, at most 255 characters; provider names
support up to 50. Merged identities resolve to their surviving agent. If an
external identity cannot be resolved, Zelto does not substitute an unrelated
agent from a matching session. See [agent identity rules](/docs/traces#identify-agents-with-your-own-ids).

For one agent per process, resource attributes are convenient. For a process
serving multiple agents, set the identity on **each span and log** instead;
record attributes override resource defaults. Do not use a room ID or version
label as the agent ID. `zelto.agent_id` and native `agentId` mean a **Zelto UUID**
and take precedence; your own UUID-shaped ID still belongs in
`zelto.agent_external_id`.

Send call data through your [provider integration](/docs/integrations) or
[call upload API](/docs/integrations/api-call-upload), using the same organization,
agent identity, provider, and external call ID. Call and trace can arrive in
either order. The trace header shows **Open conversation** when one matching
call is found; that enables the available recording, transcript, and monitor
results. See [conversation linking](/docs/traces#link-traces-to-conversations).

## Connect your existing exporter

If your application already initializes OpenTelemetry, add the Zelto exporter
to its existing provider instead of registering a second global provider.
Configure an **OTLP/HTTP** exporter; gRPC on port `4317` is not supported.

For an exporter that reads standard OpenTelemetry environment variables:

```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.zelto.ai"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20${ZELTO_INGEST_KEY}"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

The shared base endpoint has **no `/v1/traces` suffix**: the SDK appends the
signal path. A signal-specific endpoint or an explicit exporter URL is the
**full URL**, as in the Python example:

| Signal | Full URL | Accepted encoding |
| - | - | - |
| Traces | `https://ingest.zelto.ai/v1/traces` | OTLP/HTTP protobuf or OTLP JSON |
| Logs | `https://ingest.zelto.ai/v1/logs` | OTLP/HTTP protobuf or OTLP JSON |
| Native spans and logs | `https://ingest.zelto.ai/webhooks/otel` | Zelto-native JSON |

Use `Content-Type: application/x-protobuf` for protobuf and `application/json`
for JSON. Keep Python's HTTP exporter on `http/protobuf`. There is no telemetry
metrics endpoint. See the [OpenTelemetry Python exporter documentation](https://opentelemetry.io/docs/languages/python/exporters/)
for SDK configuration.

Environment variables alone do not instrument your code: initialize the SDK
and exporter, create spans (or enable your framework's instrumentation), and
end spans so they can be exported. Use a batch processor, keep it alive for the
worker's lifetime, and flush at graceful shutdown rather than after every span.
Keep trace context when work crosses asynchronous tasks or service boundaries.

### LiveKit

Use the [LiveKit tracing setup](/docs/integrations/livekit#stream-traces-opentelemetry)
at the start of the agent entrypoint, before `session.start()`. It registers
LiveKit's provider, sets your external agent identity and room reference, and
flushes pending spans on shutdown. Use the same stable agent ID and room name
as your finished-session upload. LiveKit's session instrumentation supplies the
spans; forwarding the transcript remains a separate step.

### Logs

A trace exporter does not automatically export application logs. Configure an
OTLP/HTTP **log exporter** and logging bridge for your SDK, pointed at
`https://ingest.zelto.ai/v1/logs`, with the same bearer key. Emit logs while the
relevant span is active so the bridge can attach its trace and span IDs. Include
the agent identity on the log or its resource as well. Zelto shows correlated
logs in the span's **Logs** tab and trace-wide logs in **Trace overview → Logs**.
You can also send logs with the native example below.

## Native JSON without an OpenTelemetry SDK

Use `/webhooks/otel` when your stack cannot emit OTLP. This is a different JSON
shape from OTLP JSON. Generate current timestamps and fresh identities for a
new session; save the resulting file unchanged if you need to retry:

```bash theme={null}
python3 - <<'PY' > telemetry.json
import json
import time
import uuid

now = time.time_ns() // 1_000_000
trace_id = uuid.uuid4().hex
span_id = uuid.uuid4().hex[:16]
identity = {"zelto.agent_external_id": "support-agent", "zelto.agent_provider": "zelto"}
print(json.dumps({
    "spans": [{
        "traceId": trace_id, "spanId": span_id, "name": "native.call",
        "type": "session", "startMs": now - 1000, "endMs": now,
        "status": "ok", "sessionRef": f"demo-{trace_id}",
        "attributes": identity,
    }],
    "logs": [{
        "traceId": trace_id, "spanId": span_id, "timestampMs": now,
        "severityNumber": 9, "severityText": "INFO", "body": "Session completed",
        "attributes": identity,
    }],
}))
PY
curl --fail-with-body -i https://ingest.zelto.ai/webhooks/otel \
  -H "Authorization: Bearer ${ZELTO_INGEST_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @telemetry.json
```

Expected response: HTTP `200` with `{"received":true}`. OTLP endpoints return
HTTP `200` with `{}`. Native span timestamps use **Unix milliseconds**;
OTLP timestamps use **Unix nanoseconds**. Use `parentSpanId` for native child
spans and the same `traceId` to place them in one tree. Native `attributes`
accept string, number, and boolean values.

## Limits, retries, and troubleshooting

Every telemetry request is limited to **10 MB (10,000,000 bytes) after
decompression**. Tune batch size for your span sizes; a row count alone does
not bound request bytes. Keep the original trace/span IDs, timestamps, and
payload when retrying an export. Sending the same call as new trace IDs produces
separate traces.

| Symptom | Check or action |
| - | - |
| `401` | Use an observability ingest key from the intended organization; check the bearer header and whether the key was revoked. |
| `400` | Inspect the error and correct the payload/encoding; do not repeatedly resend invalid data. Native and OTLP JSON are different schemas. |
| `413` | Split the batch below 10 MB uncompressed before retrying. |
| `429` | Honor `Retry-After` and reduce request frequency or batch more efficiently; contact Zelto for a higher organization limit. |
| `502`, `503`, `504`, timeout, or connection failure | Retain the batch and retry with exponential backoff and jitter; honor `Retry-After` when present. |
| `404` or connection to port `4317` fails | Use the HTTPS ingest host and the correct HTTP path; do not send to the dashboard or gRPC. |
| Accepted, but not visible | Allow asynchronous processing, refresh, clear filters, check organization and timestamps, and confirm the spans ended and were exported. |
| Wrong or missing agent | Check the external ID/provider pair on every span or resource, and remove an unintended explicit internal `agentId`. |
| No linked conversation | Upload the call separately with matching `call.externalId` and agent identity; check for duplicate external call references. |
| Logs absent | Configure a log exporter, not only a trace exporter; preserve trace/span context. |
| Last spans missing | Flush and shut down the provider gracefully before the process exits. |

SDK retries and in-memory queues are finite: a process crash, full queue, or
extended outage can lose telemetry **before Zelto accepts it**. Monitor exporter
errors and queue drops; use a Collector with persistent buffering if you need
durable delivery from your infrastructure. The [OTLP retry specification](https://opentelemetry.io/docs/specs/otlp/#retryable-response-codes)
describes retryable status codes.

You now have an exporter, stable agent identity, and a way to verify delivery.
Use [Traces](/docs/traces) to explore the data and the
[LiveKit integration](/docs/integrations/livekit) to attach finished calls.