> ## Documentation Index
> Fetch the complete documentation index at: https://docs.surfacearea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing and experiments

> Initialize tracing in a Node agent, wrap an LLM client, group a run into one trace, version agent builds, assign A/B variants, and let Surface Area gate which tools an agent may call.

Tracing turns a Node agent's run into a trace: one root span for the run, a generation span per model call, and a tool span per tool call. [Tracing](/tracing) explains the model; this page covers the TypeScript API.

Install the OpenTelemetry peers before using this subpath.

```bash theme={null}
npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources
```

## Initialize once at startup

`init(config?)` configures the exporter against the global OpenTelemetry provider. Call it once, before anything you want traced.

```typescript theme={null}
import { init, shutdown } from "@withgateway/sdk/tracing";

init({
  serviceName: "support-agent",
  environment: "production",
});

process.on("SIGTERM", async () => {
  await shutdown();
});
```

| Field | Type | Meaning |
| - | - | - |
| `host`, `publicKey`, `secretKey` | `string` | Credentials; fall back to the three environment variables |
| `serviceName` | `string` | Name shown on the platform; defaults to `gatewaysdk-tracing` |
| `environment` | `string` | Tag for filtering, such as `production` |
| `version` | `string` | Application version, such as a git SHA |
| `agent` | `string` | Agent id; also assembles and registers the build manifest |
| `registerBuild` | `boolean` | Set false to stamp build identity without registering it |
| `debug` | `boolean` | Log exporter activity to the console |

`isInitialized()` and `isGatewayConfigured()` report whether the call happened and whether credentials were found.

<Info>
  Importing `tracing` from `@withgateway/sdk` instead of `@withgateway/sdk/tracing` also wires experiment attributes onto every span. Use the root entry in a process that assigns variants.
</Info>

## Capture LLM calls from your client

`init()` configures the exporter. Wrap the client and every request becomes a generation span with the model, token counts, input and output.

```typescript theme={null}
import OpenAI from "openai";
import { init, observeOpenAI } from "@withgateway/sdk/tracing";

init({ serviceName: "support-agent" });

const client = observeOpenAI(new OpenAI());

const completion = await client.chat.completions.create({
  model: "gpt-5-mini",
  messages: [{ role: "user", content: "Summarize today's tickets." }],
});
```

`observeOpenAI` returns the same client, so wrap it once at startup and use it everywhere. `observeAnthropic` does the same for the Anthropic client.

```typescript theme={null}
import Anthropic from "@anthropic-ai/sdk";
import { init, observeAnthropic } from "@withgateway/sdk/tracing";

init({ serviceName: "support-agent" });
const anthropic = observeAnthropic(new Anthropic());
```

<Info>
  The wrappers use OpenInference instrumentation that ships with the package, so no extra install is needed. Apply them to the client instance your agent actually calls; a client the SDK never sees is not traced.
</Info>

`init()` removes credentials such as an MCP server's `authorization_token` (or `authorizationToken`) from every span before export, and from every property read a `customSpanProcessors` entry makes (at start, at end, or later through a kept reference); a value that names the key other than as a JSON key is replaced whole by `[redacted: holds a credential]`. The request itself is sent unchanged. On a provider you build yourself, wrap the exporting processor in `SecretRedactingSpanProcessor` from `@withgateway/sdk/tracing`.

## Group a multi-step run into one trace

`observe(name, fn, options?)` opens a span, runs the callback inside it, and closes it. Anything traced inside nests underneath, so a loop over model and tool calls reads as one trace.

```typescript theme={null}
import { observe } from "@withgateway/sdk/tracing";

const answer = await observe(
  "support-agent.run",
  async () => {
    return runToolLoop(client, prompt);   // generation and tool spans nest here
  },
  { input: prompt },
);
```

Trace a tool call the same way, with `kind: "TOOL"`. The callback's return value becomes the span's output.

```typescript theme={null}
const result = await observe("tool.search_tickets", async () => searchTickets(args), {
  kind: "TOOL",
  input: args,
});
```

| Option | Type | Meaning |
| - | - | - |
| `input` | `unknown` | Recorded as the span input, and the trace input on a root span |
| `kind` | `"AGENT" \| "CHAIN" \| "TOOL" \| "LLM" \| "RETRIEVER"` | Span kind; defaults to `AGENT` |
| `captureOutput` | `boolean` | Record the return value as the span output; defaults to true |
| `tracer` | `Tracer` | Route to an isolated provider instead of the global one |
| `attributes` | `Record<string, string>` | Extra root-span attributes, such as `gateway.session.id` |

## Attribute a run to a user

`identify(userId, options?)` attributes every span started after it in the same async context, and stamps the span that is already open so the in-flight trace is attributed immediately.

```typescript theme={null}
import { identify } from "@withgateway/sdk/tracing";

identify("u_42", {
  displayName: "Ada Lovelace",
  email: "ada@example.com",
  attributes: { plan: "enterprise" },
});
```

Passing `attributes`, `displayName` or `email` also updates the user's stored profile in the background, merging with what is already there. Profile updates need credentials; trace attribution works without them. Failures are swallowed and never produce an unhandled rejection.

## Trace a Claude Agent SDK run

The Claude Agent SDK runs the agent in a separate process, so pass its message stream through `observeClaude`. It yields the same messages while emitting an agent span with a generation per turn and a tool span per tool call.

```typescript theme={null}
import { query } from "@anthropic-ai/claude-agent-sdk";
import { init, observeClaude } from "@withgateway/sdk/tracing";

init({ serviceName: "coding-agent" });

for await (const message of observeClaude(query({ prompt, options }), {
  name: "coding-agent.run",
  input: prompt,
  model: "claude-sonnet-4-5",
})) {
  // handle messages as usual
}
```

By default the whole run is one trace. Pass `perTurn: true` and each assistant turn becomes its own trace instead, carrying that turn's generation and tool calls. Tool results feed the next turn's input, and `attributes` are stamped on every turn root.

```typescript theme={null}
for await (const message of observeClaude(stream, {
  name: "coding-agent",
  input: prompt,
  perTurn: true,
  attributes: { "gateway.session.id": runId },
})) {
  // handle messages as usual
}
```

## Trace several agents into different projects

One process can host agents that belong in different projects. `initIsolated(config?)` creates a self-contained provider with its own keys, exporter and lifecycle. It never touches the global tracer, so an app that already owns a global provider keeps it.

Spans reach an isolated provider only when they start from its `tracer`.

```typescript theme={null}
import { initIsolated, observe } from "@withgateway/sdk/tracing";

const support = initIsolated({
  publicKey: process.env.SUPPORT_PUBLIC_KEY,
  secretKey: process.env.SUPPORT_SECRET_KEY,
  serviceName: "support-agent",
  environment: "production",
});

await observe("support.run", handleTicket, {
  input: ticket,
  tracer: support.tracer,
  attributes: { "gateway.session.id": threadId, "gateway.user.id": userEmail },
});

await support.flush();
await support.shutdown();
```

A handle carries `tracer`, a `scores` client bound to the same project, `flush()` and `shutdown()`. See [Scores and signals](/sdk-ts/scores#score-into-an-isolated-project).

With the Vercel AI SDK, hand the tracer to `experimental_telemetry`. The `metadata.sessionId` and `metadata.userId` fields map to the platform's session and user automatically.

```typescript theme={null}
const result = await generateText({
  model: anthropic("claude-sonnet-4-5"),
  prompt,
  experimental_telemetry: {
    isEnabled: true,
    tracer: support.tracer,
    functionId: "support-agent",
    metadata: { sessionId: threadId, userId: userEmail },
  },
});
```

## Flush before the process exits

Spans are batched, so a short-lived process must flush or lose its last spans.

```typescript theme={null}
import { flush, shutdown } from "@withgateway/sdk/tracing";

await flush();      // export pending spans, keep tracing running
await shutdown();   // export and stop tracing
```

<Info>
  Isolated providers are not covered by the global `flush()` and `shutdown()`. Call `handle.flush()` and `handle.shutdown()` on each handle.
</Info>

## Version your agent builds

There is no build file and no lockfile. Wrap tools in `defineTool`, register code-shipped prompts, and give `init` an agent id. The SDK assembles a canonical manifest, stamps a content-addressed build hash on every span, and registers the build. Registration is idempotent, so every replica of the same code is one build.

```typescript theme={null}
import { defineTool, registerPrompt } from "@withgateway/sdk";
import { init } from "@withgateway/sdk/tracing";

const checkout = defineTool({
  name: "checkout",
  version: "2.1",              // you bump this when behavior changes
  effects: "write",            // "read" | "write" | "irreversible"
  schema: checkoutSchema,      // hashed automatically
  description: "Complete a purchase",
  execute: runCheckout,
});

const SYSTEM = registerPrompt("system", "You are a checkout assistant.");

init({ agent: "checkout-bot", version: "2.4.0" });
```

`defineTool` returns its input unchanged, so it composes with any framework. Define tools before `init`, or they do not join the build identity.

Every trace then carries `gateway.agent.id`, `gateway.agent.version` and `gateway.build.hash`. In continuous integration the actor, commit and pull request number are captured automatically as the build's attribution. `registerSkill`, `registerArtifact`, `assembleManifest` and `computeBuildHash` are exported for callers that assemble a manifest themselves.

## Assign an A/B variant

`@withgateway/sdk/experiments` assigns a stable variant for the same experiment key and user id. Initialize tracing from the root entry and spans created afterwards carry the experiment attributes.

```typescript theme={null}
import { experiments, tracing } from "@withgateway/sdk";

tracing.init({ serviceName: "checkout-agent" });

experiments.init({
  apiKey: process.env.GATEWAY_PUBLIC_KEY!,
  secretKey: process.env.GATEWAY_SECRET_KEY!,
  baseUrl: process.env.GATEWAY_HOST!,
});

await experiments.register({
  key: "checkout-prompt",
  variants: [
    { key: "control", weight: 50, config: { prompt: "Be helpful." } },
    { key: "concise", weight: 50, config: { prompt: "Be concise." } },
  ],
});

const assignment = await experiments.getVariant("checkout-prompt", "user-123");

if (assignment.inExperiment && assignment.variant) {
  console.log(assignment.variant.key, assignment.variant.config.prompt);
}
```

`getVariant` resolves to `{ variant, inExperiment, experimentKey, run }`. `variant` is null when the user is outside the experiment, and `run(fn)` executes a function inside the assignment's context so nested spans inherit it.

`experiments.fetch(keys?)` refreshes configurations from the platform, and `experiments.shutdown()` flushes pending exposures before exit.

## Let Surface Area gate an agent's tools

`@withgateway/sdk/security` decides which tools an agent may see or call, while your framework keeps owning the native tool objects. This subpath needs `@opentelemetry/api` installed.

```typescript theme={null}
import { gateway } from "@withgateway/sdk/security";

const gw = gateway.fromEnv({
  agent: { id: "support-triage", name: "Support Triage", version: "1.4.2" },
});

const tools = await gw.secureTools({
  toolsetId: "support-tools",
  provider: "zendesk",
  tools: { searchTickets, deleteTicket },
  context: { userId: "user-123", teamId: "support", environment: "production" },
});
```

`secureTools` returns the same collection with hidden tools removed and the rest wrapped. Each wrapped call is authorized before it runs. A blocked call throws `GatewayToolBlockedError`, and a call waiting on a human throws `GatewayToolApprovalRequiredError`. Security attributes are stamped on the active span when tracing is installed.

## Where to go next

* [Tracing](/tracing) for the trace model and what the platform does with it.
* [Scores and signals](/sdk-ts/scores) for attaching evaluation results to these traces.
* [Sessions and replay](/sdk-ts/replay) for reading traced sessions back.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.