Tracing and experiments

Tracing turns a Node agent’s run into a trace: one root span for the run, a generation span per model call, and a tool span per tool call. Tracing explains the model; this page covers the TypeScript API. Install the OpenTelemetry peers before using this subpath.

npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources

Initialize once at startup

init(config?) configures the exporter against the global OpenTelemetry provider. Call it once, before anything you want traced.

import { init, shutdown } from "@withgateway/sdk/tracing";

init({
  serviceName: "support-agent",
  environment: "production",
});

process.on("SIGTERM", async () => {
  await shutdown();
});

Field	Type	Meaning
`host`, `publicKey`, `secretKey`	`string`	Credentials; fall back to the three environment variables
`serviceName`	`string`	Name shown on the platform; defaults to `gatewaysdk-tracing`
`environment`	`string`	Tag for filtering, such as `production`
`version`	`string`	Application version, such as a git SHA
`agent`	`string`	Agent id; also assembles and registers the build manifest
`registerBuild`	`boolean`	Set false to stamp build identity without registering it
`debug`	`boolean`	Log exporter activity to the console

isInitialized() and isGatewayConfigured() report whether the call happened and whether credentials were found.

Importing tracing from @withgateway/sdk instead of @withgateway/sdk/tracing also wires experiment attributes onto every span. Use the root entry in a process that assigns variants.

Capture LLM calls from your client

init() configures the exporter. Wrap the client and every request becomes a generation span with the model, token counts, input and output.

import OpenAI from "openai";
import { init, observeOpenAI } from "@withgateway/sdk/tracing";

init({ serviceName: "support-agent" });

const client = observeOpenAI(new OpenAI());

const completion = await client.chat.completions.create({
  model: "gpt-5-mini",
  messages: [{ role: "user", content: "Summarize today's tickets." }],
});

observeOpenAI returns the same client, so wrap it once at startup and use it everywhere. observeAnthropic does the same for the Anthropic client.

import Anthropic from "@anthropic-ai/sdk";
import { init, observeAnthropic } from "@withgateway/sdk/tracing";

init({ serviceName: "support-agent" });
const anthropic = observeAnthropic(new Anthropic());

The wrappers use OpenInference instrumentation that ships with the package, so no extra install is needed. Apply them to the client instance your agent actually calls; a client the SDK never sees is not traced.

init() removes credentials such as an MCP server’s authorization_token (or authorizationToken) from every span before export, and from every property read a customSpanProcessors entry makes (at start, at end, or later through a kept reference); a value that names the key other than as a JSON key is replaced whole by [redacted: holds a credential]. The request itself is sent unchanged. On a provider you build yourself, wrap the exporting processor in SecretRedactingSpanProcessor from @withgateway/sdk/tracing.

Group a multi-step run into one trace

observe(name, fn, options?) opens a span, runs the callback inside it, and closes it. Anything traced inside nests underneath, so a loop over model and tool calls reads as one trace.

import { observe } from "@withgateway/sdk/tracing";

const answer = await observe(
  "support-agent.run",
  async () => {
    return runToolLoop(client, prompt);   // generation and tool spans nest here
  },
  { input: prompt },
);

Trace a tool call the same way, with kind: "TOOL". The callback’s return value becomes the span’s output.

const result = await observe("tool.search_tickets", async () => searchTickets(args), {
  kind: "TOOL",
  input: args,
});

Option	Type	Meaning
`input`	`unknown`	Recorded as the span input, and the trace input on a root span
`kind`	`"AGENT" \| "CHAIN" \| "TOOL" \| "LLM" \| "RETRIEVER"`	Span kind; defaults to `AGENT`
`captureOutput`	`boolean`	Record the return value as the span output; defaults to true
`tracer`	`Tracer`	Route to an isolated provider instead of the global one
`attributes`	`Record<string, string>`	Extra root-span attributes, such as `gateway.session.id`

Attribute a run to a user

identify(userId, options?) attributes every span started after it in the same async context, and stamps the span that is already open so the in-flight trace is attributed immediately.

import { identify } from "@withgateway/sdk/tracing";

identify("u_42", {
  displayName: "Ada Lovelace",
  email: "ada@example.com",
  attributes: { plan: "enterprise" },
});

Passing attributes, displayName or email also updates the user’s stored profile in the background, merging with what is already there. Profile updates need credentials; trace attribution works without them. Failures are swallowed and never produce an unhandled rejection.

Trace a Claude Agent SDK run

The Claude Agent SDK runs the agent in a separate process, so pass its message stream through observeClaude. It yields the same messages while emitting an agent span with a generation per turn and a tool span per tool call.

import { query } from "@anthropic-ai/claude-agent-sdk";
import { init, observeClaude } from "@withgateway/sdk/tracing";

init({ serviceName: "coding-agent" });

for await (const message of observeClaude(query({ prompt, options }), {
  name: "coding-agent.run",
  input: prompt,
  model: "claude-sonnet-4-5",
})) {
  // handle messages as usual
}

By default the whole run is one trace. Pass perTurn: true and each assistant turn becomes its own trace instead, carrying that turn’s generation and tool calls. Tool results feed the next turn’s input, and attributes are stamped on every turn root.

for await (const message of observeClaude(stream, {
  name: "coding-agent",
  input: prompt,
  perTurn: true,
  attributes: { "gateway.session.id": runId },
})) {
  // handle messages as usual
}

Trace several agents into different projects

One process can host agents that belong in different projects. initIsolated(config?) creates a self-contained provider with its own keys, exporter and lifecycle. It never touches the global tracer, so an app that already owns a global provider keeps it. Spans reach an isolated provider only when they start from its tracer.

import { initIsolated, observe } from "@withgateway/sdk/tracing";

const support = initIsolated({
  publicKey: process.env.SUPPORT_PUBLIC_KEY,
  secretKey: process.env.SUPPORT_SECRET_KEY,
  serviceName: "support-agent",
  environment: "production",
});

await observe("support.run", handleTicket, {
  input: ticket,
  tracer: support.tracer,
  attributes: { "gateway.session.id": threadId, "gateway.user.id": userEmail },
});

await support.flush();
await support.shutdown();

A handle carries tracer, a scores client bound to the same project, flush() and shutdown(). See Scores and signals. With the Vercel AI SDK, hand the tracer to experimental_telemetry. The metadata.sessionId and metadata.userId fields map to the platform’s session and user automatically.

const result = await generateText({
  model: anthropic("claude-sonnet-4-5"),
  prompt,
  experimental_telemetry: {
    isEnabled: true,
    tracer: support.tracer,
    functionId: "support-agent",
    metadata: { sessionId: threadId, userId: userEmail },
  },
});

Flush before the process exits

Spans are batched, so a short-lived process must flush or lose its last spans.

import { flush, shutdown } from "@withgateway/sdk/tracing";

await flush();      // export pending spans, keep tracing running
await shutdown();   // export and stop tracing

Isolated providers are not covered by the global flush() and shutdown(). Call handle.flush() and handle.shutdown() on each handle.

Version your agent builds

There is no build file and no lockfile. Wrap tools in defineTool, register code-shipped prompts, and give init an agent id. The SDK assembles a canonical manifest, stamps a content-addressed build hash on every span, and registers the build. Registration is idempotent, so every replica of the same code is one build.

import { defineTool, registerPrompt } from "@withgateway/sdk";
import { init } from "@withgateway/sdk/tracing";

const checkout = defineTool({
  name: "checkout",
  version: "2.1",              // you bump this when behavior changes
  effects: "write",            // "read" | "write" | "irreversible"
  schema: checkoutSchema,      // hashed automatically
  description: "Complete a purchase",
  execute: runCheckout,
});

const SYSTEM = registerPrompt("system", "You are a checkout assistant.");

init({ agent: "checkout-bot", version: "2.4.0" });

defineTool returns its input unchanged, so it composes with any framework. Define tools before init, or they do not join the build identity. Every trace then carries gateway.agent.id, gateway.agent.version and gateway.build.hash. In continuous integration the actor, commit and pull request number are captured automatically as the build’s attribution. registerSkill, registerArtifact, assembleManifest and computeBuildHash are exported for callers that assemble a manifest themselves.

Assign an A/B variant

@withgateway/sdk/experiments assigns a stable variant for the same experiment key and user id. Initialize tracing from the root entry and spans created afterwards carry the experiment attributes.

import { experiments, tracing } from "@withgateway/sdk";

tracing.init({ serviceName: "checkout-agent" });

experiments.init({
  apiKey: process.env.GATEWAY_PUBLIC_KEY!,
  secretKey: process.env.GATEWAY_SECRET_KEY!,
  baseUrl: process.env.GATEWAY_HOST!,
});

await experiments.register({
  key: "checkout-prompt",
  variants: [
    { key: "control", weight: 50, config: { prompt: "Be helpful." } },
    { key: "concise", weight: 50, config: { prompt: "Be concise." } },
  ],
});

const assignment = await experiments.getVariant("checkout-prompt", "user-123");

if (assignment.inExperiment && assignment.variant) {
  console.log(assignment.variant.key, assignment.variant.config.prompt);
}

getVariant resolves to { variant, inExperiment, experimentKey, run }. variant is null when the user is outside the experiment, and run(fn) executes a function inside the assignment’s context so nested spans inherit it. experiments.fetch(keys?) refreshes configurations from the platform, and experiments.shutdown() flushes pending exposures before exit.

Let Surface Area gate an agent’s tools

@withgateway/sdk/security decides which tools an agent may see or call, while your framework keeps owning the native tool objects. This subpath needs @opentelemetry/api installed.

import { gateway } from "@withgateway/sdk/security";

const gw = gateway.fromEnv({
  agent: { id: "support-triage", name: "Support Triage", version: "1.4.2" },
});

const tools = await gw.secureTools({
  toolsetId: "support-tools",
  provider: "zendesk",
  tools: { searchTickets, deleteTicket },
  context: { userId: "user-123", teamId: "support", environment: "production" },
});

secureTools returns the same collection with hidden tools removed and the rest wrapped. Each wrapped call is authorized before it runs. A blocked call throws GatewayToolBlockedError, and a call waiting on a human throws GatewayToolApprovalRequiredError. Security attributes are stamped on the active span when tracing is installed.

Where to go next

Tracing for the trace model and what the platform does with it.
Scores and signals for attaching evaluation results to these traces.
Sessions and replay for reading traced sessions back.

CLI

Python SDK

TypeScript SDK

SDK Reference

REST API

MCP Server

Tracing and experiments

Initialize once at startup

Capture LLM calls from your client

Group a multi-step run into one trace

Attribute a run to a user

Trace a Claude Agent SDK run

Trace several agents into different projects

Flush before the process exits

Version your agent builds

Assign an A/B variant

Let Surface Area gate an agent’s tools

Where to go next

​Initialize once at startup

​Capture LLM calls from your client

​Group a multi-step run into one trace

​Attribute a run to a user

​Trace a Claude Agent SDK run

​Trace several agents into different projects

​Flush before the process exits

​Version your agent builds

​Assign an A/B variant

​Let Surface Area gate an agent’s tools

​Where to go next

Initialize once at startup

Capture LLM calls from your client

Group a multi-step run into one trace

Attribute a run to a user

Trace a Claude Agent SDK run

Trace several agents into different projects

Flush before the process exits

Version your agent builds

Assign an A/B variant

Let Surface Area gate an agent’s tools

Where to go next