Start sending traces to Zelto

Send a test trace, find it in Zelto, then apply the same setup to your agent. A trace represents one call or session; its spans represent timed steps such as speech recognition, an LLM request, a tool call, or speech synthesis. Exporting traces does not upload a call transcript or recording: those arrive through your call integration. You need a Zelto organization, an owner or admin to create its ingest key, and a server or worker that can make HTTPS requests to ingest.zelto.ai. The first example uses Python 3.10+ and synthetic data; it does not place a call.

1. Create a telemetry key

In the organization you want to send to, open Traces → Connect agent or Settings → Integrations → Observability. Create an observability ingest key and copy it when shown. It is a write-only credential for that organization. Store it in your worker’s secrets or environment as ZELTO_INGEST_KEY. This is separate from the REST/MCP API key used to upload conversations. Telemetry uses Authorization: Bearer <ingest-key> and needs no X-Zelto-Provider header. Keep the key in server-side code.

2. Export a first trace

Create an isolated Python environment and install the SDK and HTTP exporter:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
export ZELTO_INGEST_KEY="<your-observability-ingest-key>"

Save this as send_trace.py, replacing the placeholder key in your environment before running it. This example represents one agent, so its stable identity is on the resource, the attributes shared by all spans from this provider:

import os
import uuid

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({
    "service.name": "zelto-trace-demo",
    "zelto.agent_external_id": "support-agent",
    "zelto.agent_provider": "zelto",
}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://ingest.zelto.ai/v1/traces",
    headers={"Authorization": f"Bearer {os.environ['ZELTO_INGEST_KEY']}"},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("zelto-getting-started")
call_id = f"demo-{uuid.uuid4().hex}"

try:
    with tracer.start_as_current_span("demo.call", attributes={"session.id": call_id}) as root:
        with tracer.start_as_current_span("demo.tool", attributes={
            "session.id": call_id,
            "tool.name": "check_availability",
        }):
            pass
        print(f"trace_id={root.get_span_context().trace_id:032x}")
        print(f"session.id={call_id}")
    provider.force_flush(timeout_millis=10_000)
finally:
    provider.shutdown()

python send_trace.py

You should see a trace ID and a unique session ID. The child span shares the root’s trace ID because it is created inside the active root span. Attributes such as session.id are not automatically inherited by child spans, so set them on every relevant span or through your framework’s metadata mechanism. The script flushes ended spans before exiting; inspect exporter errors too—a printed ID or a completed flush alone does not prove the server accepted it.

3. Find it in Zelto

Open Traces in the same organization, clear agent/company filters, and choose a range that includes now. Look for demo.call and compare its trace ID with the script’s output. Open it to see demo.tool beneath the root and inspect its attributes. Zelto reuses or creates the agent named support-agent. A successful HTTP response acknowledges acceptance. For organizations using buffered ingestion, data becomes queryable asynchronously; the target is within 30 seconds after acceptance under normal load, not a guaranteed maximum. Exporter batching adds time before acceptance, and processing can take longer during an outage. Refresh before diagnosing a missing trace. Linked-call panels also require the conversation to have arrived. This synthetic example has no linked conversation until a call with the same external ID is uploaded. That does not mean the trace export failed.

Identify agents and calls consistently

Identifier	Use	Lifetime
`zelto.agent_external_id`	The ID of the agent in your system; use the same value as `agent.externalId` in call uploads.	Stable across calls and releases.
`zelto.agent_provider`	The same provider namespace as call ingestion, for example `livekit`.	Stable for that integration.
`session.id`	The call, session, or room ID; match `call.externalId` in call uploads.	Unique per call; unchanged on retries.
Trace ID / span ID	OpenTelemetry identifiers generated by your SDK.	One trace per session, one span ID per operation; unchanged on retries.

Zelto matches external agent IDs within your organization. Unknown IDs create agents automatically, initially named after the external ID. If you omit the provider, Zelto reuses a unique existing match or creates a new agent under zelto. Send livekit when calls use X-Zelto-Provider: livekit to prevent separate identities when telemetry arrives first. Multiple matches across providers remain unassigned; specify the provider to disambiguate them. External IDs are case-sensitive strings, at most 255 characters; provider names support up to 50. Merged identities resolve to their surviving agent. If an external identity cannot be resolved, Zelto does not substitute an unrelated agent from a matching session. See agent identity rules. For one agent per process, resource attributes are convenient. For a process serving multiple agents, set the identity on each span and log instead; record attributes override resource defaults. Do not use a room ID or version label as the agent ID. zelto.agent_id and native agentId mean a Zelto UUID and take precedence; your own UUID-shaped ID still belongs in zelto.agent_external_id. Send call data through your provider integration or call upload API, using the same organization, agent identity, provider, and external call ID. Call and trace can arrive in either order. The trace header shows Open conversation when one matching call is found; that enables the available recording, transcript, and monitor results. See conversation linking.

Connect your existing exporter

If your application already initializes OpenTelemetry, add the Zelto exporter to its existing provider instead of registering a second global provider. Configure an OTLP/HTTP exporter; gRPC on port 4317 is not supported. For an exporter that reads standard OpenTelemetry environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.zelto.ai"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20${ZELTO_INGEST_KEY}"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

The shared base endpoint has no /v1/traces suffix: the SDK appends the signal path. A signal-specific endpoint or an explicit exporter URL is the full URL, as in the Python example:

Signal	Full URL	Accepted encoding
Traces	`https://ingest.zelto.ai/v1/traces`	OTLP/HTTP protobuf or OTLP JSON
Logs	`https://ingest.zelto.ai/v1/logs`	OTLP/HTTP protobuf or OTLP JSON
Native spans and logs	`https://ingest.zelto.ai/webhooks/otel`	Zelto-native JSON

Use Content-Type: application/x-protobuf for protobuf and application/json for JSON. Keep Python’s HTTP exporter on http/protobuf. There is no telemetry metrics endpoint. See the OpenTelemetry Python exporter documentation for SDK configuration. Environment variables alone do not instrument your code: initialize the SDK and exporter, create spans (or enable your framework’s instrumentation), and end spans so they can be exported. Use a batch processor, keep it alive for the worker’s lifetime, and flush at graceful shutdown rather than after every span. Keep trace context when work crosses asynchronous tasks or service boundaries.

LiveKit

Use the LiveKit tracing setup at the start of the agent entrypoint, before session.start(). It registers LiveKit’s provider, sets your external agent identity and room reference, and flushes pending spans on shutdown. Use the same stable agent ID and room name as your finished-session upload. LiveKit’s session instrumentation supplies the spans; forwarding the transcript remains a separate step.

Logs

A trace exporter does not automatically export application logs. Configure an OTLP/HTTP log exporter and logging bridge for your SDK, pointed at https://ingest.zelto.ai/v1/logs, with the same bearer key. Emit logs while the relevant span is active so the bridge can attach its trace and span IDs. Include the agent identity on the log or its resource as well. Zelto shows correlated logs in the span’s Logs tab and trace-wide logs in Trace overview → Logs. You can also send logs with the native example below.

Native JSON without an OpenTelemetry SDK

Use /webhooks/otel when your stack cannot emit OTLP. This is a different JSON shape from OTLP JSON. Generate current timestamps and fresh identities for a new session; save the resulting file unchanged if you need to retry:

python3 - <<'PY' > telemetry.json
import json
import time
import uuid

now = time.time_ns() // 1_000_000
trace_id = uuid.uuid4().hex
span_id = uuid.uuid4().hex[:16]
identity = {"zelto.agent_external_id": "support-agent", "zelto.agent_provider": "zelto"}
print(json.dumps({
    "spans": [{
        "traceId": trace_id, "spanId": span_id, "name": "native.call",
        "type": "session", "startMs": now - 1000, "endMs": now,
        "status": "ok", "sessionRef": f"demo-{trace_id}",
        "attributes": identity,
    }],
    "logs": [{
        "traceId": trace_id, "spanId": span_id, "timestampMs": now,
        "severityNumber": 9, "severityText": "INFO", "body": "Session completed",
        "attributes": identity,
    }],
}))
PY
curl --fail-with-body -i https://ingest.zelto.ai/webhooks/otel \
  -H "Authorization: Bearer ${ZELTO_INGEST_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @telemetry.json

Expected response: HTTP 200 with {"received":true}. OTLP endpoints return HTTP 200 with {}. Native span timestamps use Unix milliseconds; OTLP timestamps use Unix nanoseconds. Use parentSpanId for native child spans and the same traceId to place them in one tree. Native attributes accept string, number, and boolean values.

Limits, retries, and troubleshooting

Every telemetry request is limited to 10 MB (10,000,000 bytes) after decompression. Tune batch size for your span sizes; a row count alone does not bound request bytes. Keep the original trace/span IDs, timestamps, and payload when retrying an export. Sending the same call as new trace IDs produces separate traces.

Symptom	Check or action
`401`	Use an observability ingest key from the intended organization; check the bearer header and whether the key was revoked.
`400`	Inspect the error and correct the payload/encoding; do not repeatedly resend invalid data. Native and OTLP JSON are different schemas.
`413`	Split the batch below 10 MB uncompressed before retrying.
`429`	Honor `Retry-After` and reduce request frequency or batch more efficiently; contact Zelto for a higher organization limit.
`502`, `503`, `504`, timeout, or connection failure	Retain the batch and retry with exponential backoff and jitter; honor `Retry-After` when present.
`404` or connection to port `4317` fails	Use the HTTPS ingest host and the correct HTTP path; do not send to the dashboard or gRPC.
Accepted, but not visible	Allow asynchronous processing, refresh, clear filters, check organization and timestamps, and confirm the spans ended and were exported.
Wrong or missing agent	Check the external ID/provider pair on every span or resource, and remove an unintended explicit internal `agentId`.
No linked conversation	Upload the call separately with matching `call.externalId` and agent identity; check for duplicate external call references.
Logs absent	Configure a log exporter, not only a trace exporter; preserve trace/span context.
Last spans missing	Flush and shut down the provider gracefully before the process exits.

SDK retries and in-memory queues are finite: a process crash, full queue, or extended outage can lose telemetry before Zelto accepts it. Monitor exporter errors and queue drops; use a Collector with persistent buffering if you need durable delivery from your infrastructure. The OTLP retry specification describes retryable status codes. You now have an exporter, stable agent identity, and a way to verify delivery. Use Traces to explore the data and the LiveKit integration to attach finished calls.

Get started

Core concepts

Integrations

Guides

Account & settings

Legal

Start sending traces to Zelto

1. Create a telemetry key

2. Export a first trace

3. Find it in Zelto

Identify agents and calls consistently

Connect your existing exporter

LiveKit

Logs

Native JSON without an OpenTelemetry SDK

Limits, retries, and troubleshooting

​1. Create a telemetry key

​2. Export a first trace

​3. Find it in Zelto

​Identify agents and calls consistently

​Connect your existing exporter

​LiveKit

​Logs

​Native JSON without an OpenTelemetry SDK

​Limits, retries, and troubleshooting

1. Create a telemetry key

2. Export a first trace

3. Find it in Zelto

Identify agents and calls consistently

Connect your existing exporter

LiveKit

Logs

Native JSON without an OpenTelemetry SDK

Limits, retries, and troubleshooting