> ## Documentation Index
> Fetch the complete documentation index at: https://docs.surfacearea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run API worlds in CI

> A GitHub Actions workflow that runs each world's own test suite on every pull request and smoke-tests your agent against a live session, with the full YAML to copy.

The workflow below checks two things on every pull request that touches a world or the agent that calls it: whether each world still answers the way its author says it must, and whether the agent can still get through a task in one. The complete file is at [`/examples/api-worlds-ci.yml`](/examples/api-worlds-ci.yml).

## What the workflow needs

Three repository secrets, and nothing else. The keys decide which project the worlds are read from, so a CI key pointed at a staging project keeps pull requests away from production data.

| Secret | What it is |
| - | - |
| `GATEWAY_HOST` | Your Gateway URL, for example `https://withgateway.ai` |
| `GATEWAY_PUBLIC_KEY` | Project public key, `pk-lf-…` |
| `GATEWAY_SECRET_KEY` | Project secret key, `sk-lf-…` |

The `gateway` CLI reads all three straight from the environment, so no `gateway auth login` step is needed. Install it with `npm install -g @withgateway/sdk`; it needs Node 18 or newer.

<Info>
  `GATEWAY_HOST` is not secret, and a repository variable (`vars.GATEWAY_HOST`) works just as well. Keeping all three together as secrets keeps the workflow short.
</Info>

## The workflow

```yaml filename=".github/workflows/worlds.yml" theme={null}
name: worlds

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

env:
  GATEWAY_HOST: ${{ secrets.GATEWAY_HOST }}
  GATEWAY_PUBLIC_KEY: ${{ secrets.GATEWAY_PUBLIC_KEY }}
  GATEWAY_SECRET_KEY: ${{ secrets.GATEWAY_SECRET_KEY }}

jobs:
  world-tests:
    name: tests · ${{ matrix.world }}
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        world: [acme-ledger, acme-desk]
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - run: npm install -g @withgateway/sdk

      - name: Show which credentials are in effect
        run: gateway auth status

      - name: Run the world's own tests
        run: |
          gateway worlds test "${{ matrix.world }}" \
            --out "world-tests-${{ matrix.world }}.json"

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: world-tests-${{ matrix.world }}
          path: world-tests-${{ matrix.world }}.json
          if-no-files-found: ignore

  agent-smoke:
    name: agent smoke
    runs-on: ubuntu-latest
    timeout-minutes: 30
    needs: world-tests
    env:
      WORLD: acme-ledger
      TASK: default
      MIN_REWARD: "0.7"
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - run: npm install -g @withgateway/sdk

      - name: Install the repo's agent
        run: npm ci

      - name: Open a session on the world's API surface
        id: open
        run: |
          gateway worlds session open "$WORLD" -t "$TASK" -s api --no-wait > open.json
          echo "session_id=$(jq -r .sessionId open.json)" >> "$GITHUB_OUTPUT"

      - name: Wait for the session to be ready
        run: |
          for _ in $(seq 1 90); do
            gateway worlds session status "${{ steps.open.outputs.session_id }}" > session.json
            state=$(jq -r '.state // "UNKNOWN"' session.json)
            if [ "$(jq -r '.ready // false' session.json)" = "true" ]; then
              echo "ready after ${SECONDS}s"
              exit 0
            fi
            if [ "$state" = "FAILED" ] || [ "$state" = "STOPPED" ]; then
              echo "session $state before it was ready" >&2
              cat session.json >&2
              exit 1
            fi
            sleep 10
          done
          echo "session never became ready" >&2
          exit 1

      - name: Run the agent against the world
        run: |
          echo "::add-mask::$(jq -r .api.token session.json)"
          WORLD_URL="$(jq -r .api.url session.json)" \
          WORLD_TOKEN="$(jq -r .api.token session.json)" \
            node agent/run.mjs

      - name: Grade the end state
        run: |
          gateway worlds session grade "${{ steps.open.outputs.session_id }}" > grade.json
          cat grade.json
          jq -e --argjson bar "$MIN_REWARD" '(.reward // 0) >= $bar' grade.json

      - name: Close the session
        if: always()
        run: |
          gateway worlds session close "${{ steps.open.outputs.session_id }}" || true
```

## The tests job checks the worlds

`gateway worlds test <slug>` runs the suite the world's head version ships — the `tests/*.json` cases and `tests/test_*.py` files described in [Test a world](/worlds/tests) — on the platform, and exits 1 when any case fails.

One matrix entry per world means a broken world fails its own job under its own name, and `fail-fast: false` lets the rest finish, so one run reports every failure.

<Info>
  The argument is a slug, not a path. If a directory of the same name sits in the working directory, the CLI tests that directory instead. Keep matrix values distinct from your folder names, or pass `./worlds/<dir>` deliberately.
</Info>

## The smoke job checks the agent

One session, one agent, one grade. The session is opened on the world's `api` surface, the agent is pointed at `api.url` with the session token, and the task's own grader scores the end state.

Four details in that job matter.

**Open with `--no-wait` and poll.** `--no-wait` prints the session id as soon as the platform accepts the open, and the step polls `gateway worlds session status` until `ready` is true.

**Mask the token.** The session token gates the api surface, so `::add-mask::` it before it can reach a log line, and send it as an `Authorization: Bearer` header — never in a URL.

**Gate on a number.** `gateway worlds session grade` prints `{sessionId, reward, rewards, raw}`; `jq -e` turns the reward into the step's exit code so a weak run fails the job rather than passing quietly.

**Close in an `always()` step.** A session that no job closes sits there until it closes for being idle. Closing it in `always()` returns the container whether the agent passed, failed, or crashed.

Your agent reads two variables and nothing else:

```js theme={null}
// agent/run.mjs
const url = process.env.WORLD_URL;
const token = process.env.WORLD_TOKEN;

const reply = await fetch(`${url}/v1/charges?status=settled`, {
  headers: { Authorization: `Bearer ${token}` },
});
```

An agent needs a base URL and a header. Nothing else about it changes.

## Check the pull request's own tree before it is pushed

Both jobs above read published versions. A pull request that edits a world's contract, handlers or cases has not published anything yet, so add a job that tests the tree in the checkout:

```yaml theme={null}
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: gateway worlds test ./worlds/acme-ledger
```

With a `python3` of 3.12 or newer on the runner, the CLI serves the world from its bundled runtime and runs everything locally — no push, no version, no session. Install `pytest` alongside it, or the Python half of the suite fails and the `python tests` line names the install command.

## Push on merge, and let the push run the tests

Publish on merge, not on every pull request. A world whose `gateway-env.toml` carries `[actions] on_push = ["tests"]` gets its suite run against each pushed version automatically, so the merge workflow only has to push:

```yaml theme={null}
name: publish-worlds

on:
  push:
    branches: [main]

env:
  GATEWAY_HOST: ${{ secrets.GATEWAY_HOST }}
  GATEWAY_PUBLIC_KEY: ${{ secrets.GATEWAY_PUBLIC_KEY }}
  GATEWAY_SECRET_KEY: ${{ secrets.GATEWAY_SECRET_KEY }}

jobs:
  push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm install -g @withgateway/sdk
      - run: gateway bench push ./worlds/acme-ledger -m "${{ github.event.head_commit.message }}"
```

The on-push run never blocks or fails the push. Read the result afterwards in the world's Tests tab or with `gateway bench tests acme-ledger`.

<Info>
  The on-push run sees the pushed tree and nothing else, so cases that count rows fail on a version whose data has not been published yet. [Test a world](/worlds/tests#the-on-push-run-sees-the-pushed-tree-and-only-the-pushed-tree) covers how to keep JSON cases bare-tree-safe.
</Info>

## One command instead of three

`gateway worlds ci` is the whole smoke job in one line. It pushes the checked-out world, runs every task, writes a summary table into `$GITHUB_STEP_SUMMARY`, and exits 1 under the score you set:

```bash theme={null}
gateway worlds ci ./worlds/acme-ledger --agent ./agent/run.mjs --min-score 0.7
```

The branch defaults to `GITHUB_HEAD_REF` on a pull request, so each pull request's runs are tagged with its own branch. Use the longer form when you need the session, the URL, or the token yourself.

## The same gate from TypeScript

When the pull request changes the world's own source, `validateWorld` does it in one call from Node: it pushes the checkout on the pull request's branch, dispatches a named run config against that exact version, waits for every run, gates on the score, and returns a comment-ready markdown summary.

```typescript theme={null}
import { validateWorld } from "@withgateway/sdk/worlds";

const report = await validateWorld("./worlds/acme-ledger", {
  slug: "acme-ledger",
  runConfig: "pr-gate",
  minScore: 0.7,
});

process.exit(report.ok ? 0 : 1);
```

When it is your agent that changed rather than the world, `runSessions` opens one container per scenario, runs your agent against each, and gates the mean reward. Both are covered in [Gate a pull request](/sdk-ts/ci) and [Run a task suite](/sdk-ts/run-sessions).

## Where to go next

* [Test a world](/worlds/tests) for the case format and what the report says.
* [The sample repository](/worlds/sample-repo) for a working repository that runs this loop across three worlds.
* [Spin worlds up and down](/worlds/sessions) for the session lifecycle.
* [Run many sessions at once](/worlds/parallel) for running a whole scenario set instead of one smoke run.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.