gateway bench

gateway bench treats a world as a versioned bundle. Every push is content-hashed and browsable in the app, every run records the version it ran against, and any version stays addressable afterwards. Building a benchmark? A benchmark is a world with tasks; gateway benchmarks runs the worlds commands in benchmark order (init, check, push, run, results) and uses bench underneath for versions.

The product says world; the command group is still called bench and its arguments still say container. The words on screen moved to the six in the Glossary while identifiers stayed put, so existing pipelines keep working. Read container as world.

Push a version

bench push uploads the directory as a new version on a branch. The bundle is content-hashed, so pushing an unchanged tree lands on the existing version instead of creating a duplicate.

gateway bench push ./acme-world -m "add the refund route"
gateway bench push ./acme-world --branch refunds -m "wip"

Flag	What it does
`--slug`	Container slug; defaults to the directory name.
`--branch`, `-b`	Branch to push to. Defaults to the directory’s tracked branch, else `main`.
`--message`, `-m`	Change reason, shown in History.
`--author`	Attribution shown in History; defaults to `GATEWAY_AUTHOR`.
`--name`, `--description`, `--tag`, `--visibility`	Container metadata (`PRIVATE` or `PUBLIC`); `--tag` repeats.
`--auto-bump`, `--rc`, `--post`	Rewrite `[project].version` in `pyproject.toml` before hashing, so the bump lands inside the bundle.
`--image`	A pre-built OCI image reference, recorded on the version for image and agentic worlds.
`--force`	Skip the fast-forward check and move the branch even if it diverged.

A world is pushed as the tree it is: schema/world.json, connector.toml, tasks and verifiers. The platform compiles and hosts that tree, so nothing is repackaged.

Pushing onto a branch whose tip moved since you pulled is refused with a 409. Pull or checkout the new tip and push again. Use --force only when you mean to discard what moved.

Address a version with slug@ref

Most read commands take slug[@ref]. A ref resolves in four ways.

Ref form	Example	Resolves to
Branch name	`acme-world@main`	The branch’s current tip.
Tag	`acme-world@v1.2.0`	The exact version the tag points at, permanently.
Semantic version	`acme-world@0.8.0`	The version published under that `pyproject` version.
Content hash prefix	`acme-world@939cd2df`	That exact bundle.

A bare slug means the head. bench pull also accepts @latest, and bench checkout reads a bare slug as main.

gateway bench resolve acme-world@main     # pinned version id + task count
gateway bench tasks acme-world@v1.2.0     # the task list of that version

A branch moves; a tag does not. Pin a release gate or a published demo to a tag, and let CI track a branch.

Get a version onto your disk

Command	What it does
`bench pull <slug>`	Reconstruct a version’s source directory. `--target`/`-t` (or `--dir`) chooses where; the default is `./<slug>`.
`bench checkout <slug@ref>`	Pull and record the branch and base version in the directory, so later pushes fast-forward correctly.
`bench status [path]`	Where the directory stands: tracked branch, base version, whether it has local edits, whether the remote tip moved.

gateway bench checkout acme-world@main -t ./acme-world
gateway bench status ./acme-world

Use checkout when you intend to push back, pull when you only want to read a version’s files.

Pulling is for authoring. To run an agent against a world, open a hosted session by slug instead of unwrapping a copy on your own disk; see gateway worlds.

Read the history

Command	What it does
`bench log <slug>`	A branch’s version lineage, tip first: hash, semver, author, reason. `--branch`/`-b` (default `main`), `--limit`/`-n` (default 20).
`bench refs <slug>`	The container’s branches and tags.
`bench diff <base> <head>`	Per-file diff between two version ids: added, removed, modified, unchanged.

gateway bench log acme-world -b main -n 10
gateway bench diff <baseVersionId> <headVersionId>

Branch, tag and fork

Command	What it does
`bench branch [name]`	List branches and tags with the directory’s current branch marked `*`, or create a branch with `name`. `--slug`, `--path`, `--from` or `--version-id` for the start version.
`bench tag <name>`	Create an immutable tag pointing at a version. `--slug`, `--path`, `--version` or `--version-id` (default: the `main` tip).
`bench fork <slug>`	Fork a container into a new one, sharing the bundle and recording `forkedFromVersionId`. `--into` names the target slug, defaulting to `<slug>-fork`.

gateway bench branch refunds --from <versionId>
gateway bench tag v1.2.0 --version <versionId>
gateway bench fork acme-world --into acme-world-eu

Fork when a world diverges permanently — a second tenant, a different vendor plan. Branch when the change is meant to come back.

Propose a change instead of pushing one

bench propose files the directory’s local edits as a change proposal against a branch, which someone then reviews and applies. Use it where a push would be a pull request: an agent editing a world it does not own, or a change that needs a human before it reaches main.

gateway bench propose ./acme-world --title "add refunds" -m "the Sept incident needs a write path"
gateway bench proposals acme-world --status open

propose takes --title, --message/-m (aliased --reason), and --branch/-b for the branch it merges into (default main). proposals <slug> lists them and filters with --status open|applied|dismissed.

Dispatch hosted runs

A run config is a saved recipe — models, tasks, parallelism — that the platform executes on its own runners. Dispatching one from the CLI sends the same request as the Run button.

Command	What it does
`bench run-configs [action]`	`list` (default), `create` or `delete` a run config.
`bench dispatch <run_config_id>`	Start a hosted run. `--container-id` when the project hosts several; `--version-id` pins a `READY` version instead of the head.
`bench run <run_id>`	One run’s status, score and per-task rewards. `--wait` polls to terminal, bounded by `--timeout-min` (default 45).
`bench validate [path]`	Push this checkout, run the named run config against it, and gate — the pull-request entry point.
`bench secrets <action>`	`set`, `list` or `unset` the environment variables every hosted run receives. Write-only.

gateway bench run-configs create --name nightly --models openai/gpt-5-mini --parallelism 4
gateway bench dispatch <runConfigId>
gateway bench run <runId> --wait

dispatch returns as soon as the platform accepts the runs. bench run <runId> --wait polls one to terminal. run-configs create takes --name, --models (comma-separated), --parallelism (1–16), --tasks (bundled task names) or --world-tasks (stored task refs as <id> or <id>@<version>, for api worlds and not alongside --tasks), --agent for a harbor agent, and --auto-run-on-push to dispatch on every pushed READY version. delete takes --id. bench validate combines the three CI steps: --run-config <name> names the config to dispatch, --min-score fails any run below the bar, --summary-file writes a markdown summary for a pull-request comment, and --slug, --branch and --timeout-min control the rest. For a world whose agent runs locally, use gateway worlds ci instead.

gateway bench secrets set --env acme-world --key VENDOR_TOKEN   # value from stdin
gateway bench secrets list --env acme-world

Secrets are named in UPPER_SNAKE_CASE and scoped to one world with --env. Omit --value on set and the value is read from standard input, which keeps it out of shell history.

Read the tests a world ships

bench tests <slug> lists the runs of a world’s own test suite, newest first — those triggered on push, from the CLI and from the API alike.

gateway bench tests acme-world --limit 5
gateway bench tests acme-world --run <runId>

--run <id> prints one run’s full report instead of the list. Authoring and running the suite is gateway worlds test.

Commands that need the Python package

bench eval (local verifiers and harbor engines), bench install (pip-installs a version’s wheel), bench harvest-harbor (harvests a local harbor jobs directory) and bench push --build-wheel execute Python locally and belong to the Python package — see the Python SDK overview. For a hosted run, use bench dispatch.

Where to go next

gateway worlds — authoring, data, sessions and tests.
gateway benchmarks — a world with tasks, run k times per model.
The gateway CLI — credentials, the local runtime, output and exit codes.
Worlds hub — the same push and version surface from Python.

CLI

Python SDK

TypeScript SDK

SDK Reference

REST API

MCP Server

Push a version

Address a version with slug@ref

Get a version onto your disk

Read the history

Branch, tag and fork

Propose a change instead of pushing one

Dispatch hosted runs

Read the tests a world ships

Commands that need the Python package

Where to go next

​Push a version

​Address a version with slug@ref

​Get a version onto your disk

​Read the history

​Branch, tag and fork

​Propose a change instead of pushing one

​Dispatch hosted runs

​Read the tests a world ships

​Commands that need the Python package

​Where to go next

Push a version

Address a version with slug@ref

Get a version onto your disk

Read the history

Branch, tag and fork

Propose a change instead of pushing one

Dispatch hosted runs

Read the tests a world ships

Commands that need the Python package

Where to go next