Runs & Experiments

A Run groups one experiment’s work so you can watch metrics live and attach scores to the sessions or traces it produces. The run API has two equivalent styles: a context manager and a decorator. Both auto-start a run, send a heartbeat, and finish cleanly on exit or crash.

Runs require GATEWAY_PROJECT_ID. They also read GATEWAY_HOST, GATEWAY_PUBLIC_KEY, and GATEWAY_SECRET_KEY from the environment.

Runs are backend-connected. Missing credentials, a failed create request, or an invalid backend response raises gatewaysdk.GatewayRunError; the SDK does not fabricate a local Run or execute the context/decorated body as if tracking had started.

Start a run with a context manager

Use the context manager: it finishes the run even when the body raises. Pass a name and any run metadata to record.

import gatewaysdk

with gatewaysdk.run("gepa-v1", config={"lr": 0.01}, tags=["search"]) as r:
    for epoch in range(100):
        loss = train_one_epoch()
        r.log({"loss": loss}, step=epoch)

gatewaysdk.run() yields a Run. The config, labels, and tags arguments are stored with the run for reproducibility and filtering.

Start a run with a decorator

Use @gatewaysdk.experiment when an entire function is the experiment. The run starts before the function runs and finishes after it returns or raises.

import gatewaysdk

@gatewaysdk.experiment("gepa-v1", config={"lr": 0.01})
def train(run):
    run.log({"started": 1})
    for candidate in population:
        evaluate(candidate)

train()

By default the Run is injected as the first argument. Set inject_run=False to omit it and reach the run through gatewaysdk.get_current_run() instead.

@gatewaysdk.experiment(inject_run=False)
def train():
    gatewaysdk.log({"started": 1})

The decorator works on async functions too: it awaits the wrapped coroutine inside the run.

Log metrics

Call log() to record metrics. Pass a dict of name-value pairs, or a single name and value.

with gatewaysdk.run("eval") as r:
    r.log({"accuracy": 0.92, "f1": 0.88}, step=10)
    r.log("latency_ms", 240.0, step=10, unit="ms")

Metrics are buffered and flushed for live updates, so they appear on the run page while the run is still going.

Parameter	Type	Default	Description
`metrics`	`dict` or `str`	required	A `{name: value}` dict, or a single metric name
`value`	`float`	`None`	Required only when `metrics` is a single name
`step`	`int`	`None`	Training step or iteration number
`labels`	`dict`	`None`	Per-metric labels for filtering
`unit`	`str`	`""`	Unit of measurement, for example `"ms"` or `"tokens"`
`metric_type`	`str`	`"gauge"`	Visualization hint: `"gauge"`, `"counter"`, or `"histogram"`

Outside the run object, gatewaysdk.log(metrics, step=...) logs to whichever run is currently active and raises RuntimeError if none is.

Group work into episodes

An episode (a Rollout) is one interaction sequence inside a run, such as a single task attempt. Each episode gets its own session id, so the traces it produces are grouped on the Sessions dashboard.

with gatewaysdk.run("gepa-v1") as r:
    for candidate in population:
        with r.rollout(f"candidate-{candidate.id}") as episode:
            result = agent.execute(task)
            episode.score("rubric", grade(result))

Create episodes with run.rollout(name) as a context manager. The decorator form is @gatewaysdk.episode, which requires an active run created by @experiment or with gatewaysdk.run(...).

@gatewaysdk.experiment("gepa")
def train(run):
    for candidate in population:
        evaluate(candidate)

@gatewaysdk.episode(name_fn=lambda c: f"candidate-{c.id}")
def evaluate(candidate):
    result = agent.execute(task)

Pass name_fn to build the episode name from the call arguments, or name for a static one. Set inject_rollout=True to receive the Rollout as the first argument.

Post scores to sessions or traces

score() attaches an evaluation result to exactly one session or trace, so it shows up on the Scores page and in session or trace detail. Scores differ from metrics: a metric is run-level, while a score grades a specific session or trace.

with gatewaysdk.run("eval") as r:
    with r.rollout("task-1") as episode:
        result = agent.execute(task)
        episode.score("rubric", 0.85)        # targets the session; no trace required
        r.score("accuracy", 0.95, trace_id="abc123")  # or pass IDs explicitly

When you pass session_id or call score() while a Rollout is active, the session is preferred even if a trace is also available. Session-targeted scores work without a trace. Only when no session is available does score() use an explicit trace_id or fall back to the current OpenTelemetry span. If neither target is available, the score is skipped with a warning.

Parameter	Type	Default	Description
`name`	`str`	required	Score name, for example `"rubric"` or `"accuracy"`
`value`	`float` or `str`	required	Numeric for `NUMERIC`, string for `CATEGORICAL`
`trace_id`	`str`	`None`	Trace to target when no session is available; auto-detected from the active span
`session_id`	`str`	`None`	Session to target; an explicit id or active episode takes precedence over a trace
`observation_id`	`str`	`None`	Span id for a trace-targeted score; omitted from session-targeted scores
`data_type`	`str`	`"NUMERIC"`	`"NUMERIC"`, `"CATEGORICAL"`, or `"BOOLEAN"`
`comment`	`str`	`None`	Optional explanation of the score
`metadata`	`dict`	`None`	Optional metadata

Each posted score targets exactly one session or trace. observation_id applies only to trace-targeted scores.

gatewaysdk.score(name, value) is the module-level shorthand that posts to the active run.

Feed a Success Metric

Use success() for a score that should feed a dashboard Success Metric. The signal name and metric identity are separate: name must match the metric’s configured SDK signal, while metric_name must match the Success Metric name. This prevents an unrelated same-named score or signal from being counted.

with gatewaysdk.run("support-eval") as r:
    with r.rollout("ticket-42") as episode:
        resolved = agent.resolve(ticket)
        episode.success(
            "resolved_ticket",
            resolved,
            agent_id="support-triage",
            metric_name="Resolution rate",
        )

metric_name defaults to name, so you may omit it when the configured signal and Success Metric have the same name. The score follows the same target rules as score(): an explicit or active rollout session is preferred, then an explicit trace or the active OpenTelemetry trace.

Finish a run

The context manager and decorator finish the run for you. When you start a run manually with gatewaysdk.init_run(), call finish() yourself.

import gatewaysdk

r = gatewaysdk.init_run("manual-run", config={"seed": 7})
r.log({"step": 1})
r.finish(state="completed")

Valid final states are "completed", "failed", and "cancelled". gatewaysdk.finish(state) finishes the active run and does nothing if there is none. completed is confirmed only after the SDK drains every accepted metric, waits for accepted score requests, and the backend accepts the final state. If the SDK knows a metric or score was lost, or the final request is rejected or cannot be confirmed, it finishes local cleanup, sets the Run state to failed, and raises GatewayRunError instead of reporting success.

try:
    r.finish(state="completed")
except gatewaysdk.GatewayRunError as error:
    handle_delivery_failure(error)

Explicit failed and cancelled finalization remains best-effort so a secondary telemetry failure does not replace the original body exception or cancellation.

Reach the active run or episode

Two helpers return the currently active objects from anywhere in your code.

import gatewaysdk

run = gatewaysdk.get_current_run()        # Run or None
episode = gatewaysdk.get_current_rollout()  # Rollout or None

They return None when no run or episode is active, which makes logging optional in shared code.

Symbol reference

Symbol	Kind	Purpose
`run`	context manager	Start and finish a run around a block
`init_run`	function	Start a run you finish yourself
`experiment`	decorator	Wrap a function in a run
`episode`	decorator	Wrap a function in an episode within a run
`log`	function	Log metrics to the active run
`score`	function	Post a score from the active run
`finish`	function	Finish the active run
`get_current_run`	function	Return the active `Run` or `None`
`get_current_rollout`	function	Return the active `Rollout` or `None`
`GatewayRunError`	exception	Run creation or successful completion could not meet its durability contract
`Run`	class	A run; `log`, `score`, `rollout`, `finish`
`Rollout`	class	An episode; `score`, `session_id`, `trace_id`
`Metric`	dataclass	A metric with full metadata for `log_metrics`

Tracing shows how the spans that scores link to are produced.
Worlds client reads back the rollouts and per-task performance a run produces.
Evaluation & Replay covers how those scores are read against a pinned world version.

CLI

Python SDK

TypeScript SDK

SDK Reference

REST API

MCP Server

Start a run with a context manager

Start a run with a decorator

Log metrics

Group work into episodes

Post scores to sessions or traces

Feed a Success Metric

Finish a run

Reach the active run or episode

Symbol reference

​Start a run with a context manager

​Start a run with a decorator

​Log metrics

​Group work into episodes

​Post scores to sessions or traces

​Feed a Success Metric

​Finish a run

​Reach the active run or episode

​Symbol reference

​Related pages

Start a run with a context manager

Start a run with a decorator

Log metrics

Group work into episodes

Post scores to sessions or traces

Feed a Success Metric

Finish a run

Reach the active run or episode

Symbol reference

Related pages