Keep real data out of a shared world

A world built from a real vendor holds real records. A [redact] policy in connector.toml names the fields that must not travel, and decides what the world stores in their place. The policy is one block, and every surface that writes rows reads the same one. Redaction is optional. A world with no [redact] block stores every field as given, and every write path (data import, batches, session seed, a task’s seed) defaults to --redact off even when a policy exists; you opt in per write with apply or refuse. Each mode costs the world something — a search route, a join, a format — so pick per field and record the reason.

The four modes

[connector]
slug = "spycloud"
base_url = "https://api.spycloud.io"

[redact]
hash     = ["email", "full_name", "password"]
drop     = ["ssn", "results.*.raw_record"]
preserve = ["account_number", "routing_number", "phone"]
round    = { latitude = 2, longitude = 2 }

Mode	What the store holds instead	What it costs
`hash`	`"sha256:<64 hex>"`, salted with the connector’s slug	Prefix, substring and fuzzy search on the field stop working
`drop`	Nothing. The key is removed from the row	The field answers no question at all
`preserve`	A format-preserving digest: same length, digits stay digits, letters stay letters of the same case	Nothing is readable, but the value still looks like an identifier
`round`	The number rounded to the decimals you keep	Precision, and only precision

hash is the safe default. The digest is deterministic, so an exact-match lookup still resolves and a join between two entities still lines up. preserve exists because hash breaks format-constrained fields. A 15-digit account number becomes a 71-character sha256: string, and the field’s own length and format bounds then refuse the row. A shape-preserving digest stays inside the contract. Use it for identifiers — account and routing numbers, IBANs, phone numbers — and leave free text on hash, since a non-ASCII character is passed through unchanged. round keeps the numeric type and the row’s shape and drops only the precision. Coordinates are the usual case: round = { latitude = 2 } turns 37.774929 into 37.77. drop is the only mode that answers nothing. A lookup on a dropped field misses rather than errors, because the field was never written.

Both hash and preserve walk into lists and redact element by element, so an array stays an array and the world’s contract still accepts the row. Objects are left alone — name their fields instead.

Which fields a mode applies to

A bare name matches at any depth. hash = ["email"] hashes every email key anywhere in the row, however deeply nested. A dotted path matches that place only, with * standing for a list item. drop = ["results.*.raw_record"] drops raw_record from each item of the top-level results list and leaves a raw_record elsewhere in the row untouched. Use a path when a bare name would catch a key you need — a count map whose keys happen to include email, for instance. There is no entity scope. hash = ["contacts.phone"] is not “the phone of contacts”; it is a phone key under a contacts object inside a row, which no row has. A bare phone applies to every entity’s phone. gateway worlds schema check refuses a dotted path that starts with an entity name, and — in a world without connector.toml, where the policy only ever sees rows — one whose first segment is no entity’s top-level field. A bare name is never refused: it may be a vendor key the model drops before the contract sees the row, or sit inside an open nested object. gateway worlds describe shows which declared fields a policy actually lands on; a name that tags nothing there redacts nothing. round takes bare field names only, mapped to the decimals kept, 0 through 10. A field falls under exactly one mode. Naming the same field in two lists is refused when the connector is parsed, with A field is hashed, dropped, rounded or preserved, never two of those. When a name and a path could both match a value, the modes rank drop, then hash, then preserve, then round.

The salt is the connector’s slug

A hashed value is sha256 of the string "<connector slug>:<value>", written as sha256:<64 hex>. The slug is [connector] slug from connector.toml, not the world’s platform slug. The two often differ: the spycloud-world world hashes under spycloud. The plaintext is hashed exactly as given: no lowercasing, no trimming, no other salt. Applying the same rule anywhere reproduces the digest the world stored.

Computing a digest by hand with the platform slug produces a value that never matches a stored row. Read the salt from the world itself: gateway worlds describe <slug> prints it, and storedForms in the TypeScript SDK reads it from world.redact.slug.

Each world salts with its own slug, so the same email is a different digest in two worlds and nothing can be joined on disk. A link between worlds recomputes both forms from the plaintext you supply.

Where the policy runs on its own

Two paths redact without being asked:

gateway worlds connector capture redacts each response body before it is written to captures/, so a capture on disk is already redacted.
gateway worlds data extract redacts the rows it shapes, unless you pass --no-redact. Its stderr summary counts what it did: {"read": N, "rows": M, "unknownFields": {...}, "redacted": {"dropped": n, "hashed": n}}.

Every other write path stores rows exactly as given, by default. data import, a data batch and a session seed all assume the caller may already hold the rows redacted — an export of another world, or a hashed extract — because hashing a digest twice would break every join.

Captures hold real records even after redaction, because a field the policy does not name is stored in the clear. Keep captures/ out of anything you share.

Redact on the way in

Every write path takes a redact mode, and every one of them defaults to off.

Mode	What happens to a row
`off` (default)	The row lands as given, byte for byte
`apply`	The row goes through the world’s `[redact]` policy before its contract sees it — hash, drop, preserve and round, exactly as a capture would have stored it
`refuse`	A row carrying plaintext in a redacted field is refused, naming the entity, index, field and policy. Nothing is rewritten

# Rows from a real run, redacted here rather than by hand.
gateway worlds data import vendor-world rows.json --redact apply

# Check an already-redacted export before importing it.
gateway worlds data check vendor-world rows.json --redact refuse

# One live session only.
gateway worlds session seed <sessionId> rows.json --redact apply

A task can carry the mode so every session the task opens seeds the same way:

{
  "seed": {
    "mode": "append",
    "redact": "apply",
    "rows": { "people": [{ "id": "P1", "email": "jo@acme.test" }] }
  }
}

The same field reaches the other surfaces under the same name:

Surface	Where `redact` goes
REST	`redact` in the body of `POST /api/public/worlds/{slug}/data/import` and `POST /api/public/worlds/{slug}/data/batches`
MCP	`redact` on `import_world_data` and on `seed_world_session`
TypeScript SDK	`session.seed(rows, { redact })`, and `redact` on a task’s `seed` block

What `refuse` checks

refuse verifies what is verifiable from the stored value alone:

a hashed field must hold a sha256:<64 hex> digest, or a list of them;
a dropped field must be absent;
a rounded number must carry no more decimals than the policy keeps.

A preserved value is indistinguishable from a real identifier by design, so refuse lets it through. Use apply for a world with preserve fields.

apply and refuse both need the world’s policy, which lives in connector.toml. A world without one is refused up front rather than silently storing plaintext: redact=apply needs the world's [redact] policy, and this world has no connector.toml.

A refusal names the row and what was wrong with it:

people row carries plaintext in 'email' under [redact] hash: expected a sha256:<hex> digest;
import with redact=apply to redact it here, or store the redacted form

Read a redacted world back

Three surfaces report how a field is stored and what a plaintext value becomes:

gateway worlds describe <slug> stamps every field with its policy — [hash], [preserve], [round:n], [drop] — alongside entities, keys and row counts.
gateway worlds data query <world> <entity> --where email=jo@acme.test --resolve turns each value you pass into the form the store holds and runs the query on that. See Put data in a world.
describeWorld and storedForms do the same programmatically. See Describe a world.

Plan queries from describe before you write them. A field marked hash or preserve answers a plaintext lookup through resolution; a field in the clear answers it as given; a field marked drop answers nothing.

Choosing, in practice

For each field the policy touches, run the exact lookup (it must still resolve) and the search route that filters on it (it will not), then record which trade you took. Two options when a search route breaks: keep the field in plaintext because it is not sensitive, or drop the route from the world and write it into the limits. See What makes a world good.

Where to go next

Put data in a world for the rows contract and the import path.
The files a world is made of for every other connector.toml key.
Mock any vendor API for captures, handlers and conformance.
Describe a world and look things up by plaintext for the programmatic form.
Link worlds together for resolution across worlds with different salts.

Start here

Get Started

Use cases

Worlds

Evaluation & Replay

Tracing

Hooks

Keep real data out of a shared world

The four modes

Which fields a mode applies to

The salt is the connector’s slug

Where the policy runs on its own

Redact on the way in

What `refuse` checks

Read a redacted world back

Choosing, in practice

Where to go next

​The four modes

​Which fields a mode applies to

​The salt is the connector’s slug

​Where the policy runs on its own

​Redact on the way in

​What refuse checks

​Read a redacted world back

​Choosing, in practice

​Where to go next

The four modes

Which fields a mode applies to

The salt is the connector’s slug

Where the policy runs on its own

Redact on the way in

What `refuse` checks

Read a redacted world back

Choosing, in practice

Where to go next