> ## Documentation Index
> Fetch the complete documentation index at: https://docs.surfacearea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Drive a world UI

> Open a Playwright browser on a world session's dashboard, hand the browser to a model as computer-use tools, and have every action recorded on the session timeline.

A world that declares `[ui]` in its `connector.toml` serves a page for each session: the one `gateway worlds schema compile` generates from the contract (every shipped template has one), or one the world hand-writes. See [Give a world a UI](/worlds/ui). `session.browser()` opens a Playwright browser on it, either the one the platform runs next to the world or one launched locally.

Browser actions are also available as model tools, in the same shape as the world's own tools, so one agent loop drives both.

## Install Playwright

Playwright is an optional peer, loaded only when you call `session.browser()`.

```bash theme={null}
npm install playwright
npx playwright install chromium   # local mode only
```

## Open a browser on the session

Open the session asking for the surfaces you need, then open the browser.

```typescript theme={null}
import { openSession } from "@withgateway/sdk/worlds";

const session = await openSession("acme-billing@main", {
  task: "refund-double-charge",
  surfaces: ["ui", "browser"],
});
await session.ready();

const browser = await session.browser({ mode: "auto" });
await browser.goto();                 // the UI's declared entry page
await browser.click("text=Invoices");
console.log(await browser.text("main"));

await browser.close();
await session.close({ keepWarm: true });
```

| Option | Type | Default | Meaning |
| - | - | - | - |
| `mode` | `"auto" \| "hosted" \| "local"` | `"auto"` | `auto` prefers the hosted browser and falls back to a local one |
| `headless` | `boolean` | `true` | Local mode only |
| `agentId` | `string` | `GATEWAY_AGENT_ID`, else a fresh id | Whose timeline the actions land on |
| `record` | `boolean` | `true` | Set false to stop reporting actions to the platform |
| `playwright` | `PlaywrightLike` | the installed package | Inject your own Playwright module |

`mode: "hosted"` needs `surfaces: ["browser"]` and `mode: "local"` needs `surfaces: ["ui"]`. Asking for a mode the session cannot serve throws `WorldSurfaceUnavailable`.

<Info>
  Several agents may share one session and one hosted browser. Each reports under its own `agentId`, so the session page shows each agent's timeline separately, and reopening with the same id continues its numbering rather than restarting at zero.
</Info>

## Every browser method

| Method | Returns | What it does |
| - | - | - |
| `goto(pathOrUrl?)` | `Promise<string>` | Opens a UI path such as `/customers`, a full URL, or the entry page |
| `back()` | `Promise<string>` | One page back in history |
| `click(selector)` | `Promise<void>` | Clicks by Playwright selector |
| `type(selector, text)` | `Promise<void>` | Replaces an input's value |
| `press(key)` | `Promise<void>` | Presses a key such as `Enter` or `Tab` |
| `scroll(dy)` | `Promise<void>` | Scrolls vertically; negative scrolls up |
| `screenshot(fullPage?)` | `Promise<Buffer>` | A PNG |
| `screenshotBase64(fullPage?)` | `Promise<string>` | The same PNG, base64 encoded |
| `text(selector?)` | `Promise<string>` | Visible text, whole page by default |
| `html()` | `Promise<string>` | The page's HTML |
| `title()` | `Promise<string>` | The page title |
| `close()` | `Promise<void>` | Closes the browser and flushes recorded actions |

`browser.url` is the current URL, `browser.mode` is the resolved mode, `browser.viewport` is the size in use, and `browser.page` is the underlying Playwright page for direct Playwright calls.

Selectors are Playwright's own, so `text=Refund` and `role=button[name=Save]` work alongside CSS.

## When the hosted browser is reclaimed

A World Host browser on the `standard` tier can be reclaimed mid-session; the world is not. When a call fails because the browser is gone, a hosted browser asks the platform for a new one, reconnects within 90 seconds and reopens the page it was on. A read (`text()`, `screenshot()`, `title()`, `html()`) or a `goto()` is then retried once. A `click()`, `type()`, `press()`, `back()` or `scroll()` is not replayed, because it may already have reached the world: it throws `BrowserReconnected` (with `outcome` and `restoredUrl`) so the agent looks at the page before acting again. The timeline shows one `browser_reconnect` event. Driving your own Playwright? Call `session.repairBrowser()` after a dropped connection: it answers `healthy` or `replaced` (reconnect to the same `wsUrl`) or `starting` (ask again after `retryAfterMs`).

```ts theme={null}
const { outcome, retryAfterMs } = await session.repairBrowser();
// { outcome: "replaced", retryAfterMs: null }
```

## Hand the browser to a model

`session.computerUseTools(browser)` returns a `WorldToolkit` of eight browser actions, in the same shape `session.toolkit()` uses. Merge the two and one agent loop drives both the world's tools and its screen.

```typescript theme={null}
const worldTools = await session.toolkit();
const screenTools = session.computerUseTools(browser);

const schemas = [...worldTools.schemas, ...screenTools.schemas];
const impls = { ...worldTools.impls, ...screenTools.impls };

const result = await impls.browser_goto({ path: "/invoices" });
console.log(result); // { url, title, text, truncated }
```

| Tool | Arguments | Returns |
| - | - | - |
| `browser_goto` | `path` optional | The URL, title and page text |
| `browser_click` | `selector` | The page text after the click |
| `browser_type` | `selector`, `text` | `{ ok, url }` |
| `browser_press` | `key` | The page text after |
| `browser_scroll` | `dy`, default 600 | `{ ok, url }` |
| `browser_screenshot` | `full_page`, default false | `{ image_base64, url }` |
| `browser_read` | `selector`, default `body` | The URL, title and page text |
| `browser_back` | none | The page text after |

Page text is truncated at 20,000 characters, and the result says `truncated: true` when it was.

To give Claude or an OpenAI model its own pixel-based computer-use tool instead, use `session.nativeComputer(browser, "anthropic" | "openai")`: see [Native computer use](/worlds/computer-use).

## Actions are on the record

Every action reports what was asked, whether it worked, and a screenshot of the screen afterwards. A failed action is reported the same way, with the screen the agent saw.

Arguments are redacted before they leave the process. Values that look like passwords, tokens, API keys or authorization headers are stripped, both by field name and by value shape.

`session.browserRecorder(agentId?)` returns the same reporter for a Playwright page you drive yourself.

## Where to go next

* [World sessions](/sdk-ts/sessions) for the session the browser attaches to.
* [Run a task suite](/sdk-ts/run-sessions) to run UI tasks in parallel.
* [Give a world a UI](/worlds/ui) for declaring, generating or hand-writing the page.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.