Observe your agent

Ratel emits telemetry on two deliberately separate streams (ADR 0007):

• Local trace events, owned by the Rust core. Structured events (search, invoke_start, upstream_error, ...) recorded into a sink you configure; this stream feeds the offline inspector, Ratel Local statusline and savings reporting, and future rerankers. Its reliability contract is query-log semantics: best-effort, sampleable, lossy on backpressure. Losing an event is acceptable, corrupting a catalog is not.
• OpenTelemetry. Remote telemetry is OpenTelemetry, on two signals: gen_ai.* spans per a pinned semconv baseline plus a ratel.* overlay, and — when content capture opts in — structured Logs EventRecords carrying the gated content to the sibling /v1/logs endpoint. Both are stock OTLP (http/protobuf + Bearer auth). This is the stream for Ratel Cloud, Langfuse, or your own collector.

Both are off by default: the local sink is noop until you pass one, and spans and EventRecords are no-ops until OpenTelemetry providers exist. Pick your path:

• Local visibility, no network → configure a trace sink.
• No OpenTelemetry in your stack yet → greenfield setup: Python calls configure_telemetry once at startup; TypeScript hosts build and register their own providers — the SDK ships no bootstrap.
• Already running OpenTelemetry (Langfuse, the Vercel AI SDK, your own collector) → your provider already receives the spans. Dual-export the Ratel cut: ratel_span_processor + ratel_log_record_processor in Python, a filtered processor you build yourself in TypeScript.

Local trace sinks

Pass trace at construction — ratel({ trace }) in TypeScript forwards it to both internal catalogs; Python passes it to the ToolCatalog (or SkillCatalog) constructor — and the core records every search, invoke, capability-tool, upstream-MCP, and auth event into the sink.

from ratel_ai import ToolCatalog, TraceSinkConfig

catalog = ToolCatalog(
    trace=TraceSinkConfig(kind="jsonl", session_id="session-1", path="/tmp/ratel.jsonl"),
)
# Every catalog.invoke, capability-tool call, and register_mcp_server call now
# appends one JSON line per event to /tmp/ratel.jsonl.

import { ratel } from "@ratel-ai/sdk";

const r = ratel({
  trace: { kind: "jsonl", sessionId: "session-1", path: "/tmp/ratel.jsonl" },
});
// Every invoke, capability-tool call, and registerMcpServer call now appends one JSON line per event to /tmp/ratel.jsonl.

Every recorded event lands wrapped in an envelope: { "v": 1, "ts": <unix epoch ms>, "session_id": "...", "type": "search", ...event fields }, with the event type tagged in snake_case.

• Draining the memory sink. catalog.drain_trace_events() (Python) / r.tools.catalog.drainTraceEvents() (TypeScript — the drain lives on the raw ToolCatalog under the r.tools handle) returns the buffered events and removes them. It returns [] unless the active sink is memory; the jsonl and noop sinks have no drain, read the file instead.
• JSONL file behavior. The sink creates missing parent directories, opens the file append-only with mode 0600 on Unix, and flushes after every line. Writes are best-effort: a serialization or write failure is swallowed rather than crashing the agent loop.
• Origin tagging. Searches the model makes through search_capabilities are tagged origin: "agent"; direct catalog.search(query, k) calls (in TypeScript, on the raw r.tools.catalog) default to "direct", overridable with a third argument.

Event types

The event set is defined once, in the Rust core, so every SDK emits the same shapes. The full type vocabulary:

A single agent call often records several of these: search_capabilities produces both a core-level search (or skill_search) and a gateway_search, and invoke_tool produces invoke_start / invoke_end (or invoke_error) plus gateway_invoke / gateway_error, with upstream_invoke / upstream_error on top for proxied MCP tools. The four auth_* events come only from Ratel Local OAuth upstreams, which embeds the SDK and records into the same stream; the SDKs never emit them.

The OpenTelemetry span vocabulary

Independently of the local sink, the SDK emits OpenTelemetry spans around the same funnel: gen_ai.* attributes pinned to semconv v1.42.0, plus the Ratel-owned ratel.* overlay. The local event and the span are two independent channels around the same call.

Tool invocations deliberately reuse execute_tool, the standard gen_ai.operation.name value, instead of a bespoke ratel.invoke span, so a generic OTel backend already understands them. Errors use the standard span status ERROR plus the exception event, not a bespoke attribute. The SDK does not emit the LLM-call spans (chat <model>): those come from your LLM instrumentation, and Ratel's spans join the same trace next to them.

No-op until a provider

Instrumentation is free until you opt in, in both languages:

• TypeScript: the SDK's only OpenTelemetry dependencies are the API-only @opentelemetry/api and @opentelemetry/api-logs, never the OTel SDK. Until the host registers providers, every span is a no-op NonRecordingSpan, every EventRecord is dropped, and the local trace stream is untouched.
• Python: the base ratel-ai install depends only on ratel-ai-telemetry>=0.1.3, the OTel-free vocabulary — still no OpenTelemetry. If opentelemetry is missing, every telemetry helper is a straight pass-through, zero overhead, no spans. With it installed, spans and EventRecords go to whatever providers are registered.

Greenfield: ship spans to Ratel Cloud

No OpenTelemetry in your stack yet? In Python, one call at startup registers global tracer and logger providers with batch OTLP exporters pointed at Ratel Cloud. TypeScript has no Ratel bootstrap: the host builds and registers the providers itself.

pip install 'ratel-ai[otlp]'

import os
from ratel_ai import configure_telemetry

handle = configure_telemetry(
    endpoint=os.environ.get("RATEL_OTLP_ENDPOINT", "https://cloud.ratel.sh/v1/traces"),
    api_key=os.environ["RATEL_API_KEY"],
)
# ... run your agent: every ratel.* / execute_tool span (and gated EventRecord) now exports ...
handle.shutdown()  # flush both exporters on exit (handle.force_flush() mid-run)

npm install @opentelemetry/sdk-node @opentelemetry/sdk-trace-base @opentelemetry/sdk-logs @opentelemetry/exporter-trace-otlp-proto @opentelemetry/exporter-logs-otlp-proto

import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-proto";

const endpoint = process.env.RATEL_OTLP_ENDPOINT ?? "https://cloud.ratel.sh/v1/traces";
const headers = { Authorization: `Bearer ${process.env.RATEL_API_KEY}` };

const sdk = new NodeSDK({
  spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({ url: endpoint, headers }))],
  logRecordProcessors: [
    new BatchLogRecordProcessor({
      exporter: new OTLPLogExporter({ url: endpoint.replace("/v1/traces", "/v1/logs"), headers }),
    }),
  ],
});
sdk.start();
// ... run your agent: every ratel.* / execute_tool span (and gated EventRecord) now exports ...
await sdk.shutdown(); // flush stays with you — the host owns the providers

Endpoint and auth resolution is Python-only — in TypeScript the host owns both entirely (as above):

• The endpoint= option wins; otherwise RATEL_OTLP_ENDPOINT; otherwise the superseded RATEL_URL, which still works but raises a DeprecationWarning (it also names the catalog source, so it no longer doubles as the OTLP destination). Either way it is the full OTLP traces URL including /v1/traces. With none set, the call fails before anything registers.
• The Logs exporter derives its URL by swapping /v1/traces for /v1/logs; logs_endpoint= overrides it.
• The API key is sent as Authorization: Bearer <key> on top of any extra headers. When omitted it falls back to the RATEL_API_KEY env var (unless you already set an Authorization header explicitly).
• service_name defaults to "ratel".

Runnable examples live in the repo: examples/telemetry-ts and examples/telemetry-python emit a sample ratel.search + execute_tool trace to a console exporter, no collector or API key needed, and switch to a real OTLP export when RATEL_OTLP_ENDPOINT is set — the TypeScript one also emits a ratel.search.results EventRecord through a LoggerProvider.

Already on OpenTelemetry: dual-export

OpenTelemetry's coexistence model is one provider per signal, many processors. If Langfuse, the Vercel AI SDK, or your own collector already owns the providers, the SDK's telemetry is already flowing to them; add processors that send a filtered copy to Ratel Cloud. Python ships them packaged; in TypeScript you build your own:

import os

from opentelemetry import _logs, trace
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk.trace import TracerProvider
from ratel_ai_telemetry.otlp import ratel_log_record_processor, ratel_span_processor

endpoint = os.environ.get("RATEL_OTLP_ENDPOINT", "https://cloud.ratel.sh/v1/traces")
api_key = os.environ["RATEL_API_KEY"]

provider = trace.get_tracer_provider()  # the provider your existing setup registered
if not isinstance(provider, TracerProvider):
    raise RuntimeError("register your OpenTelemetry SDK provider before adding processors")
provider.add_span_processor(ratel_span_processor(endpoint=endpoint, api_key=api_key))

# The Logs half — without it, captured content never reaches Cloud.
logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(
    ratel_log_record_processor(endpoint=endpoint, api_key=api_key)
)
_logs.set_logger_provider(logger_provider)

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { BatchSpanProcessor, type ReadableSpan } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { LangfuseSpanProcessor } from "@langfuse/otel";

const ratelSignal = (span: ReadableSpan) =>
  span.name.startsWith("ratel.") ||
  Object.keys(span.attributes).some((k) => k.startsWith("gen_ai.") || k.startsWith("ratel."));

class RatelSpanProcessor extends BatchSpanProcessor {
  onEnd(span: ReadableSpan) {
    if (ratelSignal(span)) super.onEnd(span);
  }
}

const provider = new NodeTracerProvider({
  spanProcessors: [
    new LangfuseSpanProcessor(), // Langfuse keeps every span
    new RatelSpanProcessor(      // Ratel takes gen_ai.*/ratel.* only
      new OTLPTraceExporter({
        url: process.env.RATEL_OTLP_ENDPOINT ?? "https://cloud.ratel.sh/v1/traces",
        headers: { Authorization: `Bearer ${process.env.RATEL_API_KEY}` },
      }),
    ),
  ],
});
provider.register();

The filter is the point — forward a span if its name starts with ratel. or any attribute key starts with gen_ai. or ratel., so the framework's ai.* wrapper noise stays out:

• In Python that predicate is the packaged default of ratel_span_processor; ratel_log_record_processor's default ratel_event_filter does the same for EventRecords, forwarding only those named gen_ai.* / ratel.*. In TypeScript you own the predicate (above).
• Pass span_filter=lambda _s: True (Python) — or drop the filter from your own processor (TypeScript) — to forward everything. Do that, or tail-sample, when full-trace fidelity matters: per-span filtering can orphan the AI SDK's ai.* wrapper span from its gen_ai.* child.
• Python's packaged exporters carry no resource; your provider keeps ownership of service.name.

Content capture

Message and tool content never leaves the process by default. Set the mode in code (configure_telemetry(capture_content=...) in Python; setContentCapture(mode) in TypeScript, re-exported from @ratel-ai/sdk) or through the ecosystem instrumentation flag OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, the fallback when nothing is set in code; parsing is trimmed and case-insensitive.

The four modes select two independent channels:

• Span attributes (SPAN_ONLY / SPAN_AND_EVENT): ratel.search.query on ratel.search, gen_ai.tool.call.arguments / gen_ai.tool.call.result on execute_tool.
• Logs EventRecords (EVENT_ONLY / SPAN_AND_EVENT): a ratel.tool.execution.details record carrying structured gen_ai.tool.call.arguments and, on success, gen_ai.tool.call.result — the Python SDK records only the stable MCP CallToolResult fields (content, structuredContent, isError) — plus a ratel.search.results record carrying ratel.search.query. These need a registered Logs provider; EVENT_ONLY still puts nothing on span attributes.

The gen_ai.client.inference.operation.details event stays your LLM instrumentation's convention for message content (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages). Output messages require a finish_reason, so a tool result is never encoded there.

Next steps