# tapir:observe Return a structured candidate list of every interactive widget on screen. The command mirrors the Stagehand observe-once-act-many pattern: the agent observes once, then issues many `tapir:tap` / `tapir:type` / `tapir:drag` calls against the minted `qN` refs without re-observing between actions. No LLM is invoked server-side. The handler walks the live Semantics tree, mints a re-resolvable `qN` handle for each interactive widget, and returns the candidate list as JSON. The agent reads the list and decides which refs to act on. This is what differentiates `tapir:observe` from a model-side `tapir:snap`: it returns a flat, role-filterable list optimised for LLM consumption rather than the full tree. A control wrapped in `MergeSemantics` is listed once, as its boundary node: the descendants it absorbs are never sent to the platform and get no candidate and no `qN` handle. The CLI surface is mostly for debugging; the MCP descriptor is the primary surface for agent integrations. - [Synopsis](#synopsis) - [Arguments](#arguments) - [Returns](#returns) - [Observe-once-act-many](#observe-once-act-many) - [Examples](#examples) - [Related Documentation](#related-documentation) ## Synopsis ``` dart run tapir tapir:observe [--intent=] [--roles=] [--limit=] [--includeEnrichers=] ``` `tapir:observe` requires a running Flutter session (`CommandBoot.connected`). It dials the VM Service URI, calls `ext.tapir.observe`, and prints the JSON candidate list to stdout. --- ## Arguments | Option | Type | Default | Description | |--------|------|---------|-------------| | `intent` | string | unset | Free-form caller hint describing what the agent is looking for. Accepted for caller-side logging only; the handler does not use it for ranking or filtering and the response does not carry it. | | `roles` | csv string | unset (every role) | Comma-separated role filter (e.g. `button,textbox,checkbox`). Omit for every role. Useful when the agent already knows it only cares about, say, form fields. | | `limit` | int (string) | `50` | Maximum number of candidates to return. The handler walks the tree in order and stops once it has N. | | `includeEnrichers` | enum string | `true` | One of `true` (default; `magicFormField`, `magicRoute`, `magicGateResult` and the `wind` fields `breakpoint` and `states`), `false` (no enricher fields), `full` (every enricher field, including every `wind` sub-field). Use `full` when the agent needs the complete wind block; use `false` for the smallest payload. | All four options pass through to the VM Service handler as string values (no client-side parsing). Empty strings are dropped so the handler sees absent rather than empty when the caller omits an option. --- ## Returns The VM Service handler returns a JSON envelope; the CLI dumps it to stdout via `jsonEncode`. **Success envelope (illustrative; `includeEnrichers=true`, single candidate shown):** ```json { "candidates": [ { "ref": "q1", "role": "button", "label": "Sign in", "value": null, "bounds": {"x": 120, "y": 400, "w": 240, "h": 48}, "isEnabled": true, "isVisible": true, "inViewport": true, "occluded": false, "magicRoute": "/login", "wind": {"breakpoint": "md", "states": "hover"} } ], "count": 1 } ``` Every candidate ships with: - A re-resolvable `qN` handle (Playwright Locator semantics: every action call re-walks the tree). - The Semantics `role` (button, textbox, checkbox, link, etc.). - The Semantics `label` (visible text or explicit a11y label) and its `value`, `null` when empty. - The widget `bounds` as `{x, y, w, h}`. - `isEnabled`, `isVisible` (non-empty bounds that overlap the viewport), `inViewport` (`false` when the bounds lie outside the view) and `occluded` (`true` when a hit test at the bounds' centre reaches a widget painted above the candidate instead of the candidate itself; only probed for a visible candidate, so `false` for one that is not). - `tapOwner`, `{x, y, w, h}`, only on a merged candidate (a `MergeSemantics` boundary) whose bounds' centre is not on its own control: the control's rect, which a `tapir:tap` on the candidate goes to unless `--no-retarget` is passed (#60); `centreHandled: true` means the centre has a tap handler of its own. Absent on every other candidate. - The enricher fields (`magicFormField`, `magicRoute`, `magicGateResult`, `wind`) as top-level keys when `includeEnrichers` is `true` or `full` and an enricher produced them. **Error envelope:** The VM Service handler propagates errors as `ServiceExtensionResponse.error(extensionError, message)`. Common causes: no running app at the recorded URI, `TapirPlugin.install()` never called. --- ## Observe-once-act-many The Stagehand pattern that gives `tapir:observe` its name: 1. **Observe once.** A single `tapir:observe` call enumerates the interactive surface of the current screen, mints `qN` handles, and returns them in one JSON payload. 2. **Act many.** The agent issues a sequence of `tapir:tap --ref=qN`, `tapir:type --ref=qN`, `tapir:set_checkbox --ref=qN`, etc. against the minted refs WITHOUT re-observing between actions. Each action re-resolves the `qN` handle against the live tree, so the refs survive intermediate rebuilds. Candidates sharing one label stay distinct through that re-resolution. Each handle carries its position among the nodes that label matched, which lets a list of rows offering the same action be worked row by row while the list is unchanged; a handle reports stale only when fewer same-label nodes remain than its position. `tapir:find` is deliberately different, because there the predicate is one you chose: it answers the first match and reports the total so you can refine it or scope it with `--within`. The index is a position and not an identity, so it survives neither a row leaving nor a row arriving above it. A row inserted above the list between the observe call and the action (a pull-to-refresh, a lazily built header, a banner carrying the same label) shifts every held handle down by one, and a row removed above it shifts them up by one, silently, because nothing re-checks the node it lands on. Playwright's `nth(i)` behaves the same way. Re-observe after anything that can add or remove a row with that label; only a handle whose position no longer exists is reported stale. The "no server-side LLM" property is the second half of the pattern: Stagehand-the-product runs an LLM server-side to rank candidates by intent. `tapir:observe` returns the raw candidate list and lets the agent's own LLM rank, so no model context is consumed on the server, and the response is deterministic. Re-observe only when: - The agent navigated to a new screen (the handles minted on the previous screen become stale matches). - The candidate set itself changes (e.g. a modal opens, a list grows or shrinks, a tab switches). - Same-label rows are reordered (a sort, a filter, a drag to reorder): the held position then selects a different row without any candidate being added or removed. For incremental state changes on the same screen (clicking a button that disables another button, typing into a field that reveals a new form section), re-resolution on every action call is sufficient; no second `tapir:observe` is needed. --- ## Examples ### 1. Enumerate every interactive widget on the current screen ```bash dart run tapir tapir:observe ``` Returns up to 50 candidates with the default subset of enricher fields. Useful as the first call after a navigation to discover what is on screen. ### 2. Filter to a single role ```bash dart run tapir tapir:observe --roles=button --limit=10 ``` Limits the response to up to 10 button candidates. Useful when the agent already knows the next action is a tap. ### 3. Observe followed by act-many ```bash dart run tapir tapir:observe --roles=textbox,button > /tmp/observe.json # agent reads /tmp/observe.json, decides to type into q1 then tap q2 dart run tapir tapir:type --ref=q1 --text="user@example.com" dart run tapir tapir:tap --ref=q2 ``` No re-observe between the two actions; both `qN` handles re-resolve against the live tree on each call. --- ## Related Documentation - [tapir:snap](https://anilcancakir.com/tapir/docs/commands/tapir-snap.md): the raw Semantics-tree YAML; richer than `tapir:observe` but with `eN` refs that go stale on rebuild. - [tapir:find](https://anilcancakir.com/tapir/docs/commands/tapir-find.md): mint a single `qN` handle from a known predicate; pair with `tapir:observe` when the agent already knows what to look for. - [tapir:tap](https://anilcancakir.com/tapir/docs/commands/tapir-tap.md), `tapir:type`, `tapir:drag`: the action commands that consume the `qN` refs minted by `tapir:observe`.