Run API worlds in CI

The workflow below checks two things on every pull request that touches a world or the agent that calls it: whether each world still answers the way its author says it must, and whether the agent can still get through a task in one. The complete file is at /examples/api-worlds-ci.yml.

What the workflow needs

Three repository secrets, and nothing else. The keys decide which project the worlds are read from, so a CI key pointed at a staging project keeps pull requests away from production data.

Secret	What it is
`GATEWAY_HOST`	Your Gateway URL, for example `https://withgateway.ai`
`GATEWAY_PUBLIC_KEY`	Project public key, `pk-lf-…`
`GATEWAY_SECRET_KEY`	Project secret key, `sk-lf-…`

The gateway CLI reads all three straight from the environment, so no gateway auth login step is needed. Install it with npm install -g @withgateway/sdk; it needs Node 18 or newer.

GATEWAY_HOST is not secret, and a repository variable (vars.GATEWAY_HOST) works just as well. Keeping all three together as secrets keeps the workflow short.

The workflow

name: worlds

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

env:
  GATEWAY_HOST: ${{ secrets.GATEWAY_HOST }}
  GATEWAY_PUBLIC_KEY: ${{ secrets.GATEWAY_PUBLIC_KEY }}
  GATEWAY_SECRET_KEY: ${{ secrets.GATEWAY_SECRET_KEY }}

jobs:
  world-tests:
    name: tests · ${{ matrix.world }}
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        world: [acme-ledger, acme-desk]
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - run: npm install -g @withgateway/sdk

      - name: Show which credentials are in effect
        run: gateway auth status

      - name: Run the world's own tests
        run: |
          gateway worlds test "${{ matrix.world }}" \
            --out "world-tests-${{ matrix.world }}.json"

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: world-tests-${{ matrix.world }}
          path: world-tests-${{ matrix.world }}.json
          if-no-files-found: ignore

  agent-smoke:
    name: agent smoke
    runs-on: ubuntu-latest
    timeout-minutes: 30
    needs: world-tests
    env:
      WORLD: acme-ledger
      TASK: default
      MIN_REWARD: "0.7"
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - run: npm install -g @withgateway/sdk

      - name: Install the repo's agent
        run: npm ci

      - name: Open a session on the world's API surface
        id: open
        run: |
          gateway worlds session open "$WORLD" -t "$TASK" -s api --no-wait > open.json
          echo "session_id=$(jq -r .sessionId open.json)" >> "$GITHUB_OUTPUT"

      - name: Wait for the session to be ready
        run: |
          for _ in $(seq 1 90); do
            gateway worlds session status "${{ steps.open.outputs.session_id }}" > session.json
            state=$(jq -r '.state // "UNKNOWN"' session.json)
            if [ "$(jq -r '.ready // false' session.json)" = "true" ]; then
              echo "ready after ${SECONDS}s"
              exit 0
            fi
            if [ "$state" = "FAILED" ] || [ "$state" = "STOPPED" ]; then
              echo "session $state before it was ready" >&2
              cat session.json >&2
              exit 1
            fi
            sleep 10
          done
          echo "session never became ready" >&2
          exit 1

      - name: Run the agent against the world
        run: |
          echo "::add-mask::$(jq -r .api.token session.json)"
          WORLD_URL="$(jq -r .api.url session.json)" \
          WORLD_TOKEN="$(jq -r .api.token session.json)" \
            node agent/run.mjs

      - name: Grade the end state
        run: |
          gateway worlds session grade "${{ steps.open.outputs.session_id }}" > grade.json
          cat grade.json
          jq -e --argjson bar "$MIN_REWARD" '(.reward // 0) >= $bar' grade.json

      - name: Close the session
        if: always()
        run: |
          gateway worlds session close "${{ steps.open.outputs.session_id }}" || true

The tests job checks the worlds

gateway worlds test <slug> runs the suite the world’s head version ships — the tests/*.json cases and tests/test_*.py files described in Test a world — on the platform, and exits 1 when any case fails. One matrix entry per world means a broken world fails its own job under its own name, and fail-fast: false lets the rest finish, so one run reports every failure.

The argument is a slug, not a path. If a directory of the same name sits in the working directory, the CLI tests that directory instead. Keep matrix values distinct from your folder names, or pass ./worlds/<dir> deliberately.

The smoke job checks the agent

One session, one agent, one grade. The session is opened on the world’s api surface, the agent is pointed at api.url with the session token, and the task’s own grader scores the end state. Four details in that job matter. Open with --no-wait and poll. --no-wait prints the session id as soon as the platform accepts the open, and the step polls gateway worlds session status until ready is true. Mask the token. The session token gates the api surface, so ::add-mask:: it before it can reach a log line, and send it as an Authorization: Bearer header — never in a URL. Gate on a number. gateway worlds session grade prints {sessionId, reward, rewards, raw}; jq -e turns the reward into the step’s exit code so a weak run fails the job rather than passing quietly. Close in an always() step. A session that no job closes sits there until it closes for being idle. Closing it in always() returns the container whether the agent passed, failed, or crashed. Your agent reads two variables and nothing else:

// agent/run.mjs
const url = process.env.WORLD_URL;
const token = process.env.WORLD_TOKEN;

const reply = await fetch(`${url}/v1/charges?status=settled`, {
  headers: { Authorization: `Bearer ${token}` },
});

An agent needs a base URL and a header. Nothing else about it changes.

Check the pull request’s own tree before it is pushed

Both jobs above read published versions. A pull request that edits a world’s contract, handlers or cases has not published anything yet, so add a job that tests the tree in the checkout:

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: gateway worlds test ./worlds/acme-ledger

With a python3 of 3.12 or newer on the runner, the CLI serves the world from its bundled runtime and runs everything locally — no push, no version, no session. Install pytest alongside it, or the Python half of the suite fails and the python tests line names the install command.

Push on merge, and let the push run the tests

Publish on merge, not on every pull request. A world whose gateway-env.toml carries [actions] on_push = ["tests"] gets its suite run against each pushed version automatically, so the merge workflow only has to push:

name: publish-worlds

on:
  push:
    branches: [main]

env:
  GATEWAY_HOST: ${{ secrets.GATEWAY_HOST }}
  GATEWAY_PUBLIC_KEY: ${{ secrets.GATEWAY_PUBLIC_KEY }}
  GATEWAY_SECRET_KEY: ${{ secrets.GATEWAY_SECRET_KEY }}

jobs:
  push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm install -g @withgateway/sdk
      - run: gateway bench push ./worlds/acme-ledger -m "${{ github.event.head_commit.message }}"

The on-push run never blocks or fails the push. Read the result afterwards in the world’s Tests tab or with gateway bench tests acme-ledger.

The on-push run sees the pushed tree and nothing else, so cases that count rows fail on a version whose data has not been published yet. Test a world covers how to keep JSON cases bare-tree-safe.

One command instead of three

gateway worlds ci is the whole smoke job in one line. It pushes the checked-out world, runs every task, writes a summary table into $GITHUB_STEP_SUMMARY, and exits 1 under the score you set:

gateway worlds ci ./worlds/acme-ledger --agent ./agent/run.mjs --min-score 0.7

The branch defaults to GITHUB_HEAD_REF on a pull request, so each pull request’s runs are tagged with its own branch. Use the longer form when you need the session, the URL, or the token yourself.

The same gate from TypeScript

When the pull request changes the world’s own source, validateWorld does it in one call from Node: it pushes the checkout on the pull request’s branch, dispatches a named run config against that exact version, waits for every run, gates on the score, and returns a comment-ready markdown summary.

import { validateWorld } from "@withgateway/sdk/worlds";

const report = await validateWorld("./worlds/acme-ledger", {
  slug: "acme-ledger",
  runConfig: "pr-gate",
  minScore: 0.7,
});

process.exit(report.ok ? 0 : 1);

When it is your agent that changed rather than the world, runSessions opens one container per scenario, runs your agent against each, and gates the mean reward. Both are covered in Gate a pull request and Run a task suite.

Where to go next

Test a world for the case format and what the report says.
The sample repository for a working repository that runs this loop across three worlds.
Spin worlds up and down for the session lifecycle.
Run many sessions at once for running a whole scenario set instead of one smoke run.

Start here

Get Started

Use cases

Worlds

Evaluation & Replay

Tracing

Hooks

What the workflow needs

The workflow

The tests job checks the worlds

The smoke job checks the agent

Check the pull request’s own tree before it is pushed

Push on merge, and let the push run the tests

One command instead of three

The same gate from TypeScript

Where to go next

​What the workflow needs

​The workflow

​The tests job checks the worlds

​The smoke job checks the agent

​Check the pull request’s own tree before it is pushed

​Push on merge, and let the push run the tests

​One command instead of three

​The same gate from TypeScript

​Where to go next

What the workflow needs

The workflow

The tests job checks the worlds

The smoke job checks the agent

Check the pull request’s own tree before it is pushed

Push on merge, and let the push run the tests

One command instead of three

The same gate from TypeScript

Where to go next