Skip to main content
Seven properties separate a world you can trust from a mock that agrees with everything: real evidence, real writes, measured parity, faithful failures, honest redaction, versioning, and written-down limits. Each section below ends with a check you can run.

​
Seed from evidence, never from imagination

Every row in a world should trace back to a capture from the vendor’s API, an export from the system of record, or a ledger of production calls. Invented rows encode your assumptions about the data, and the agent then passes by agreeing with them. gateway worlds connector capture records what the live service returns, page by page, redacted, with the secret never written to disk. gateway worlds connector ingest seeds the world from those captures through its own contract, so a row that does not fit the schema is refused rather than reshaped. Record where each field came from. schema/provenance.json holds the source URLs, retrieval dates and hashes of the bytes you pulled, and captures/manifest.json carries a sha256 per captured page.
The check. Pick any entity and ask where its rows came from. Every one should answer with a capture file, an export, or a labelled synthetic fixture, and the third answer should be rare and marked as synthetic.

​
Serve writes, not just lookups

A read-only mock tests almost nothing. The decisions that hurt in production happen when the agent creates something, gets a 422 back, retries, hits a 409 because the state moved, and has to work out what to do. A world that only answers GET exercises none of it. Writes belong in the world’s handlers, where the vendor’s rules live: the fields it validates, the state machine it enforces, the ownership it derives from the bearer token. A write that persists through the world’s contract means the next read sees it, and a body the vendor would reject gets the vendor’s rejection.
The check. Count the operations in connector.toml by action. If every one is read, the world is a lookup table. Then take the call log of a real client — gateway worlds session export <sessionId> --requests, which includes the requests it sent straight to the world’s routes — and confirm each write it makes has a route that persists it and a rule that can refuse it.
If the vendor returns 422 for a missing field and your world returns 201, your agent has been taught that malformed requests succeed.

​
Measure parity instead of asserting it

Measure parity against the client’s own traffic: take the calls a real client makes, replay them against the world, and compare the status, the envelope and the counts. gateway worlds connector conform ./world --openapi vendor.yaml calls every declared operation on the served world and checks each answer against the vendor’s published response schema. gateway worlds session export gives the real ledger: one JSONL line per call, with args and result. Turn that ledger into tests — one described case per route the client hits. The Tests tab shows coverage grouped by route family, with a passed count per group.
The check. List the distinct routes in the client’s call ledger. Every one should appear as at least one case in tests/ with a description saying why. See Test a world.

​
Reproduce the failures, not only the successes

Agents spend their worst moments in error handling, so the errors have to be the vendor’s. A 401 with the wrong body shape, a 404 where the vendor returns an empty list, a 403 where the vendor uses 401 — each one trains your agent on a service that does not exist. Capture records non-2xx replies, because the vendor’s error bodies are part of the contract. connector.toml carries the shapes: [api] unauthorized for the body, [api] unauthorized_status for vendors that answer 403 rather than 401. The token flow is declared the same way, so a client that performs an OAuth2 grant before its first call works against the world unchanged.
The check. Write one case per failure mode with auth: false, and assert the body, not just the status. Then take one route where the vendor returns an empty result for an unknown id and confirm the world does not return 404.

​
Redact in a way that keeps the shape

Redaction that changes behaviour is a bug. [redact] hash replaces a value with a stable hash; [redact] drop removes the field entirely; [redact] preserve keeps the value’s length and character classes so a format-constrained identifier still validates; [redact] round keeps a number and drops its precision. All four are in Keep real data out. Hashing is the safer default because it is deterministic: the same input hashes to the same output, so exact-match lookups still resolve and joins between entities still line up. Prefix, substring and fuzzy search are the trade-off. A hashed email cannot be matched by its domain, and a hashed name cannot be found by its first three letters, so any route whose semantics are match rather than equality stops behaving like the vendor’s. Two options: keep that field in plaintext because it is not sensitive, or drop the route from the world and write it into the limits.
The check. For each hashed field, run the exact lookup (it must still resolve) and the search route that filters on it (it will not). Then write down which of the two you chose and why.
Captures hold real records even after redaction. Keep captures/ out of anything you share — the contract, connector.toml and the handlers are what travel between projects.

​
Version everything, and pin every run to a version

A world that moves under you makes its own results meaningless. Every gateway bench push is a version, every run records the version it faced, and a session can be pinned to an exact one with --version-id.
Tests belong to the version too. [actions] on_push = ["tests"] in gateway-env.toml runs the suite against each pushed version, so which versions were green is a fact on the record.
The check. Take a result from a month ago, find the version id it ran against, and open a session on that exact version. If you cannot, the result cannot be reproduced.

​
Write down what the world does not do

Write the world’s edges where the next person will read them. An undocumented gap gets discovered as a confusing agent failure rather than as a known limitation. Three kinds recur:
  • Unmodelled surfaces. Webhooks, streaming endpoints, file uploads, and admin consoles the agent never touches but a reader might assume are there.
  • Computed routes. Aggregates, analytics and scoring endpoints whose real values come from a pipeline you did not replicate. Serve an honest 501 rather than a plausible number.
  • Paging over unions. A vendor that pages across several underlying collections in one response rarely has an order your store reproduces exactly; say whether order is guaranteed.
The check. List every route in the client’s ledger. Each one should be either implemented in connector.toml or named in the limits with one line saying why not.
Never claim measured fidelity you did not measure. A check that could not run is skipped, not passed, and a synthetic fixture is synthetic no matter how realistic it looks.

​
Definition of done

A world is ready to be run against when all of the following are true.
  • Every entity’s rows trace to a capture, an export, or a fixture explicitly labelled synthetic.
  • schema/provenance.json records source, retrieval date and hash for each pinned source.
  • Every write the client makes has a route that persists it and a rule that can refuse it.
  • gateway worlds connector conform --openapi <spec> exits zero, or every remaining mismatch is in the limits.
  • Every route in the client’s call ledger has at least one case in tests/ carrying a description.
  • gateway worlds test ./<world> passes locally, with pytest installed so the Python half runs.
  • The vendor’s 401, 403 and 404 bodies are asserted, not just their status codes.
  • Each redacted field is recorded with the search behaviour it costs.
  • The world is pushed, [actions] on_push = ["tests"] is set, and the latest version’s run is green.
  • Limits are written down: unmodelled surfaces, computed routes, paging guarantees.

​
Where to go next