## The rows file
Pick one of three shapes per file.
**An object of entity to rows.** Several entities in one file.
```json theme={null}
{
"users": [{ "id": "U1", "name": "Ada", "email": "ada@example.com" }],
"channels": [{ "id": "C1", "name": "general", "created_by": "U1" }]
}
```
**JSON Lines, one row per line, each naming its entity.**
```jsonl theme={null}
{"entity": "users", "row": {"id": "U1", "name": "Ada"}}
{"entity": "channels", "row": {"id": "C1", "name": "general", "created_by": "U1"}}
```
**Bare rows with `--entity`.** A JSON array or JSON Lines of plain objects, for one entity.
```bash theme={null}
gateway worlds data import ./vendor-world users.jsonl --entity users
```
Entity names are the world's own: the entities in `schema/world.json`. Fields keep the
vendor's casing.
## Append or replace
* **`--mode append`** (the default) adds rows. A row whose primary key already exists
updates that row rather than duplicating it.
* **`--mode replace`** removes every row of each entity the file names, then inserts the
file's rows. Entities the file does not name are untouched.
The whole file is checked first. One bad row and nothing is written; the refusal points at
the row and the field, such as `/users/3/email`.
## 1. See what the world holds
```bash theme={null}
gateway worlds data show ./vendor-world
gateway worlds data show vendor-world --entity users # a platform world, one entity's rows
```
The first line of the output is the world's kind. An import lands as a contract-checked batch
in `data/initial.json`.
### Ask for the rows you mean
`data show --entity` dumps an entity. `data query` filters it, pages it, and finds a redacted
row from the plaintext you know:
```bash theme={null}
gateway worlds describe vendor-world # what is there, and how each field is stored
gateway worlds data query vendor-world users --where plan=pro --limit 50
gateway worlds data query vendor-world users --where email=jo@acme.test --resolve
gateway worlds data query vendor-world users --where id=1 --where id=2 # a repeated key is IN
```
`--where k=v` is equality on the stored value, repeatable; the same key twice means "either".
`--limit` and `--offset` page. Rows go to stdout as one JSON object per line, so the output
pipes into `jq`; the `n of total` footer and the `resolved` lines go to stderr. Without `--where`,
`data query` is `data show --entity` with a page size.
A world built from a vendor hashes its identifying fields on disk, so
`--where email=jo@acme.test` alone matches nothing. With `--resolve`, each value you pass is
turned into the form the store holds, per the world's `[redact]` policy, and the query runs on
that. A field the policy drops resolves to nothing, and the output says so, because that field
was never written. `worlds describe` shows which fields resolve, and to what.
The same query is `GET /api/public/worlds/{slug}/data?entity=&where=&resolve=true`, `queryRows`
in [the TypeScript SDK](/sdk-ts/describe), and the `query_world_data` tool for an agent. A world
whose rows live in a data snapshot answers it too, without a session.
### Ask in SQL
`data sql` runs one read-only SQL statement over a schema world's rows, whatever database the
world uses. Use it for joins, counts and `GROUP BY` that `data query` cannot express:
```bash theme={null}
gateway worlds data sql acme-world 'SELECT id, name FROM users LIMIT 5'
gateway worlds data sql ./acme-world 'SELECT plan, COUNT(*) AS n FROM users GROUP BY plan' --json
```
One statement only, and only reads: a write, `ATTACH`, `PRAGMA` or a second statement is
refused, and so is a statement that runs past 10 s, makes a value over 4 MiB or answers more
than 16 MiB. A blob comes back as `{"base64": "..."}`, and a float too large for JSON as the
text `"inf"` or `"-inf"`. At most 1000 rows come back (`--limit`); the output says when there were more. The
same read is `POST /api/public/worlds/{slug}/data/sql` with `{"sql": "..."}` and the
`sql_world_data` tool for an agent. A world built on `db/schema.sql` is a plain SQLite file:
read it with `sqlite3`.
## 2. Import
```bash theme={null}
gateway worlds data import ./vendor-world rows.json
gateway worlds data import vendor-world rows.json --message "seed Q3 tenants"
gateway worlds data import vendor-world rows.json --publish later
```
Give a local directory and `data/initial.json` is updated in place.
Give a platform world's slug and the rows stream up as gzipped chunks into a **data batch**.
The platform validates every row against the contract, applies the batch onto the world's
snapshot, and turns it into a data-only version. `--publish now` (the default) cuts that
version per batch; `--publish later` leaves the batch pending until you fold several together
with `data publish`.
Imported rows stay with the world as you keep building it. Add a tool, change a handler, a
route, a test or the docs, and the next version serves the same rows with no new import, from a
sandbox or a local checkout alike, and a fork of the world (`gateway bench fork`) starts with the
same rows. A change to the data's shape (an entity, a field or its description, a key or a
relationship) needs the rows imported again against the new shape, and `schema compile` names
what changed.
Exit 0 means applied or published, exit 2 means refused with a report written, exit 1 is an
error.
### Check before you write
```bash theme={null}
gateway worlds data check vendor-world rows.json --report refusals.jsonl
```
`data check` is `data import --dry-run` under its own name: every row is validated against
the world's contract and nothing is uploaded or written. A slug runs a dry-run batch on the
platform; a directory runs the import over a scratch copy. Exit 2 when any row is refused,
and `--report` writes every refusal as JSON Lines rather than only the first few.
### Redact on the way in
Imported rows are stored exactly as given. Pass `--redact` when they are not already redacted:
```bash theme={null}
gateway worlds data import vendor-world rows.json --redact apply
gateway worlds data check vendor-world rows.json --redact refuse
```
`apply` runs the world's `[redact]` policy over every row first; `refuse` rejects a row that
carries plaintext in a redacted field. Both need a `connector.toml`. See
[Keep real data out](/worlds/redaction).
` |
| `default = ...` | used when `from` is missing or null; without one the field is not set |
| `type = "string" \| "integer" \| "number" \| "boolean"` | coerces; a CSV cell is text until this says otherwise, and a cell that cannot be coerced refuses the record |
| `{ literal = ... }` | a constant |
| `{ template = "deleted contact {id}" }` | `{path}` holes filled from the item or the record; an empty hole refuses the record |
| `{ hash = ["field", "$tool"] }` | the sha256 of those values (a JSON array), for an id the source does not carry; unset when one is missing |
The file is checked when it loads: every entity and field must exist in the contract, every
transform must be in the set, every `carry` must name a field the parent maps. `[*]` walks a
list in `select` and `explode.path`; `select = "result[*]"` reads an array result item by
item. `at = "updated_at"` on a source dates its rows from the item for `on_conflict`;
otherwise the record's `at` does.
```bash theme={null}
gateway worlds data ingest acme-crm calls.jsonl --map ingest.toml --dry-run
gateway worlds data ingest acme-crm saved.json --from rows --map ingest.toml # JSON objects; a saved result is one record, select rows[*]
gateway worlds data ingest acme-crm deals.csv --from csv --map ingest.toml # header row; every cell is text
gateway worlds data ingest acme-crm calls.jsonl # the world's own ingest.toml, when it has one
```
The dry run reports per source: records seen, items selected, rows per entity, **unmapped
source fields** with a sample value, and **entity fields never set**; then the unmatched
records by name and every refusal with its record index.
```text theme={null}
13 records, 10 ingested, 1 skipped, 0 refused (dry run) — via map
accounts: 3 rows
activities: 3 rows
contacts: 3 rows
deals: 2 rows
skipped: call failed 1
unmatched (no [[sources]] claimed them, skipped): crm_read 1
source crm_read, crm_update_account (tool-call, select result) -> accounts: 2 records, 2 items, rows accounts 2, contacts 1, deals 1
unmapped source fields: error ("not found")
source crm_delete_contact (tool-call, select args) -> activities: 1 records, 1 items, rows activities 1
activities fields never set: account_id, deal_id
entity counts: accounts 3, activities 3, contacts 3, deals 2
```
`map init` writes the file to start from: one `[[sources]]` per source shape in a sample
(per tool for tool calls), every source field with a sample value on one side, every entity
field on the other. Only an exact-name match is filled in, and each one is marked `# exact-name
match — confirm`; every other field is a commented `# entity.field = ""` line.
```bash theme={null}
gateway worlds data ingest map init ./acme-crm calls.jsonl # writes ./acme-crm/ingest.toml
gateway worlds data ingest map init acme-crm saved.json --from rows --entity accounts --out -
```
### Tool calls that already are the operation's response
With neither a transform nor a mapping, and no `ingest.toml` in the world, a tool-call record
goes through its tool's `[[operations]]` entry in `connector.toml`: the `result` is read at the
operation's `results` path and projected by its `[ingest]` table (`drop`, `explode`,
`carry_args` from the call's `args`) exactly as [a capture](/worlds/custom-connections) is.
Use it when a tool's result is the vendor's response body and nothing else. The report says
`via operations` and notes that this is the operation's projection, not a mapping; a tool that
is not one of the world's operations is refused by name.
```bash theme={null}
gateway worlds data ingest ./vendor-world calls.jsonl
gateway worlds data ingest vendor-world --from traces --session