Drive a world UI

A world that declares [ui] in its connector.toml serves a page for each session: the one gateway worlds schema compile generates from the contract (every shipped template has one), or one the world hand-writes. See Give a world a UI. session.browser() opens a Playwright browser on it, either the one the platform runs next to the world or one launched locally. Browser actions are also available as model tools, in the same shape as the world’s own tools, so one agent loop drives both.

Install Playwright

Playwright is an optional peer, loaded only when you call session.browser().

npm install playwright
npx playwright install chromium   # local mode only

Open a browser on the session

Open the session asking for the surfaces you need, then open the browser.

import { openSession } from "@withgateway/sdk/worlds";

const session = await openSession("acme-billing@main", {
  task: "refund-double-charge",
  surfaces: ["ui", "browser"],
});
await session.ready();

const browser = await session.browser({ mode: "auto" });
await browser.goto();                 // the UI's declared entry page
await browser.click("text=Invoices");
console.log(await browser.text("main"));

await browser.close();
await session.close({ keepWarm: true });

Option	Type	Default	Meaning
`mode`	`"auto" \| "hosted" \| "local"`	`"auto"`	`auto` prefers the hosted browser and falls back to a local one
`headless`	`boolean`	`true`	Local mode only
`agentId`	`string`	`GATEWAY_AGENT_ID`, else a fresh id	Whose timeline the actions land on
`record`	`boolean`	`true`	Set false to stop reporting actions to the platform
`playwright`	`PlaywrightLike`	the installed package	Inject your own Playwright module

mode: "hosted" needs surfaces: ["browser"] and mode: "local" needs surfaces: ["ui"]. Asking for a mode the session cannot serve throws WorldSurfaceUnavailable.

Several agents may share one session and one hosted browser. Each reports under its own agentId, so the session page shows each agent’s timeline separately, and reopening with the same id continues its numbering rather than restarting at zero.

Every browser method

Method	Returns	What it does
`goto(pathOrUrl?)`	`Promise<string>`	Opens a UI path such as `/customers`, a full URL, or the entry page
`back()`	`Promise<string>`	One page back in history
`click(selector)`	`Promise<void>`	Clicks by Playwright selector
`type(selector, text)`	`Promise<void>`	Replaces an input’s value
`press(key)`	`Promise<void>`	Presses a key such as `Enter` or `Tab`
`scroll(dy)`	`Promise<void>`	Scrolls vertically; negative scrolls up
`screenshot(fullPage?)`	`Promise<Buffer>`	A PNG
`screenshotBase64(fullPage?)`	`Promise<string>`	The same PNG, base64 encoded
`text(selector?)`	`Promise<string>`	Visible text, whole page by default
`html()`	`Promise<string>`	The page’s HTML
`title()`	`Promise<string>`	The page title
`close()`	`Promise<void>`	Closes the browser and flushes recorded actions

browser.url is the current URL, browser.mode is the resolved mode, browser.viewport is the size in use, and browser.page is the underlying Playwright page for direct Playwright calls. Selectors are Playwright’s own, so text=Refund and role=button[name=Save] work alongside CSS.

When the hosted browser is reclaimed

A World Host browser on the standard tier can be reclaimed mid-session; the world is not. When a call fails because the browser is gone, a hosted browser asks the platform for a new one, reconnects within 90 seconds and reopens the page it was on. A read (text(), screenshot(), title(), html()) or a goto() is then retried once. A click(), type(), press(), back() or scroll() is not replayed, because it may already have reached the world: it throws BrowserReconnected (with outcome and restoredUrl) so the agent looks at the page before acting again. The timeline shows one browser_reconnect event. Driving your own Playwright? Call session.repairBrowser() after a dropped connection: it answers healthy or replaced (reconnect to the same wsUrl) or starting (ask again after retryAfterMs).

const { outcome, retryAfterMs } = await session.repairBrowser();
// { outcome: "replaced", retryAfterMs: null }

Hand the browser to a model

session.computerUseTools(browser) returns a WorldToolkit of eight browser actions, in the same shape session.toolkit() uses. Merge the two and one agent loop drives both the world’s tools and its screen.

const worldTools = await session.toolkit();
const screenTools = session.computerUseTools(browser);

const schemas = [...worldTools.schemas, ...screenTools.schemas];
const impls = { ...worldTools.impls, ...screenTools.impls };

const result = await impls.browser_goto({ path: "/invoices" });
console.log(result); // { url, title, text, truncated }

Tool	Arguments	Returns
`browser_goto`	`path` optional	The URL, title and page text
`browser_click`	`selector`	The page text after the click
`browser_type`	`selector`, `text`	`{ ok, url }`
`browser_press`	`key`	The page text after
`browser_scroll`	`dy`, default 600	`{ ok, url }`
`browser_screenshot`	`full_page`, default false	`{ image_base64, url }`
`browser_read`	`selector`, default `body`	The URL, title and page text
`browser_back`	none	The page text after

Page text is truncated at 20,000 characters, and the result says truncated: true when it was. To give Claude or an OpenAI model its own pixel-based computer-use tool instead, use session.nativeComputer(browser, "anthropic" | "openai"): see Native computer use.

Actions are on the record

Every action reports what was asked, whether it worked, and a screenshot of the screen afterwards. A failed action is reported the same way, with the screen the agent saw. Arguments are redacted before they leave the process. Values that look like passwords, tokens, API keys or authorization headers are stripped, both by field name and by value shape. session.browserRecorder(agentId?) returns the same reporter for a Playwright page you drive yourself.

Where to go next

World sessions for the session the browser attaches to.
Run a task suite to run UI tasks in parallel.
Give a world a UI for declaring, generating or hand-writing the page.

CLI

Python SDK

TypeScript SDK

SDK Reference

REST API

MCP Server

Install Playwright

Open a browser on the session

Every browser method

When the hosted browser is reclaimed

Hand the browser to a model

Actions are on the record

Where to go next

​Install Playwright

​Open a browser on the session

​Every browser method

​When the hosted browser is reclaimed

​Hand the browser to a model

​Actions are on the record

​Where to go next

Install Playwright

Open a browser on the session

Every browser method

When the hosted browser is reclaimed

Hand the browser to a model

Actions are on the record

Where to go next