tapir:observe

The command mirrors the Stagehand observe-once-act-many pattern: the agent observes once, then issues many tapir:tap / tapir:type / tapir:drag calls against the minted qN refs without re-observing between actions.

No LLM is invoked server-side. The handler walks the live Semantics tree, mints a re-resolvable qN handle for each interactive widget, and returns the candidate list as JSON. The agent reads the list and decides which refs to act on. This is what differentiates tapir:observe from a model-side tapir:snap: it returns a flat, role-filterable list optimised for LLM consumption rather than the full tree.

A control wrapped in MergeSemantics is listed once, as its boundary node: the descendants it absorbs are never sent to the platform and get no candidate and no qN handle.

The CLI surface is mostly for debugging; the MCP descriptor is the primary surface for agent integrations.

Synopsis#

dart run tapir tapir:observe [--intent=<hint>]
                                       [--roles=<csv>]
                                       [--limit=<n>]
                                       [--includeEnrichers=<true|false|full>]

tapir:observe requires a running Flutter session (CommandBoot.connected). It dials the VM Service URI, calls ext.tapir.observe, and prints the JSON candidate list to stdout.

Arguments#

All four options pass through to the VM Service handler as string values (no client-side parsing). Empty strings are dropped so the handler sees absent rather than empty when the caller omits an option.

Returns#

The VM Service handler returns a JSON envelope; the CLI dumps it to stdout via jsonEncode.

Success envelope (illustrative; includeEnrichers=true, single candidate shown):

{
  "candidates": [
    {
      "ref": "q1",
      "role": "button",
      "label": "Sign in",
      "value": null,
      "bounds": {"x": 120, "y": 400, "w": 240, "h": 48},
      "isEnabled": true,
      "isVisible": true,
      "inViewport": true,
      "occluded": false,
      "magicRoute": "/login",
      "wind": {"breakpoint": "md", "states": "hover"}
    }
  ],
  "count": 1
}

Every candidate ships with:

• A re-resolvable qN handle (Playwright Locator semantics: every action call re-walks the tree).
• The Semantics role (button, textbox, checkbox, link, etc.).
• The Semantics label (visible text or explicit a11y label) and its value, null when empty.
• The widget bounds as {x, y, w, h}.
• isEnabled, isVisible (non-empty bounds that overlap the viewport), inViewport (false when the bounds lie outside the view) and occluded (true when a hit test at the bounds' centre reaches a widget painted above the candidate instead of the candidate itself; only probed for a visible candidate, so false for one that is not).
• tapOwner, {x, y, w, h}, only on a merged candidate (a MergeSemantics boundary) whose bounds' centre is not on its own control: the control's rect, which a tapir:tap on the candidate goes to unless --no-retarget is passed (#60); centreHandled: true means the centre has a tap handler of its own. Absent on every other candidate.
• The enricher fields (magicFormField, magicRoute, magicGateResult, wind) as top-level keys when includeEnrichers is true or full and an enricher produced them.

Error envelope:

The VM Service handler propagates errors as ServiceExtensionResponse.error(extensionError, message). Common causes: no running app at the recorded URI, TapirPlugin.install() never called.

Observe-once-act-many#

The Stagehand pattern that gives tapir:observe its name:

1. Observe once. A single tapir:observe call enumerates the interactive surface of the current screen, mints qN handles, and returns them in one JSON payload.
2. Act many. The agent issues a sequence of tapir:tap --ref=qN, tapir:type --ref=qN, tapir:set_checkbox --ref=qN, etc. against the minted refs WITHOUT re-observing between actions. Each action re-resolves the qN handle against the live tree, so the refs survive intermediate rebuilds.

Candidates sharing one label stay distinct through that re-resolution. Each handle carries its position among the nodes that label matched, which lets a list of rows offering the same action be worked row by row while the list is unchanged; a handle reports stale only when fewer same-label nodes remain than its position. tapir:find is deliberately different, because there the predicate is one you chose: it answers the first match and reports the total so you can refine it or scope it with --within.

The index is a position and not an identity, so it survives neither a row leaving nor a row arriving above it. A row inserted above the list between the observe call and the action (a pull-to-refresh, a lazily built header, a banner carrying the same label) shifts every held handle down by one, and a row removed above it shifts them up by one, silently, because nothing re-checks the node it lands on. Playwright's nth(i) behaves the same way. Re-observe after anything that can add or remove a row with that label; only a handle whose position no longer exists is reported stale.

The "no server-side LLM" property is the second half of the pattern: Stagehand-the-product runs an LLM server-side to rank candidates by intent. tapir:observe returns the raw candidate list and lets the agent's own LLM rank, so no model context is consumed on the server, and the response is deterministic.

Re-observe only when:

• The agent navigated to a new screen (the handles minted on the previous screen become stale matches).
• The candidate set itself changes (e.g. a modal opens, a list grows or shrinks, a tab switches).
• Same-label rows are reordered (a sort, a filter, a drag to reorder): the held position then selects a different row without any candidate being added or removed.

For incremental state changes on the same screen (clicking a button that disables another button, typing into a field that reveals a new form section), re-resolution on every action call is sufficient; no second tapir:observe is needed.

Examples#

1. Enumerate every interactive widget on the current screen#

dart run tapir tapir:observe

Returns up to 50 candidates with the default subset of enricher fields. Useful as the first call after a navigation to discover what is on screen.

2. Filter to a single role#

dart run tapir tapir:observe --roles=button --limit=10

Limits the response to up to 10 button candidates. Useful when the agent already knows the next action is a tap.

3. Observe followed by act-many#

dart run tapir tapir:observe --roles=textbox,button > /tmp/observe.json
# agent reads /tmp/observe.json, decides to type into q1 then tap q2
dart run tapir tapir:type --ref=q1 --text="[email protected]"
dart run tapir tapir:tap --ref=q2

No re-observe between the two actions; both qN handles re-resolve against the live tree on each call.

Related Documentation#

• tapir:snap: the raw Semantics-tree YAML; richer than tapir:observe but with eN refs that go stale on rebuild.
• tapir:find: mint a single qN handle from a known predicate; pair with tapir:observe when the agent already knows what to look for.
• tapir:tap, tapir:type, tapir:drag: the action commands that consume the qN refs minted by tapir:observe.