Embodied systems · Explore this field ↗ · Implementation · 4 min read
Multimodal grounding and coordinate frames: show what the system means by there
Images, screenshots, and spatial interfaces need an explicit reference frame, freshness check, and confirmation path before a vague visual interpretation becomes an action.
Grounded visual instruction
- 01Capture source
- 02Name frame
- 03Find semantic target
- 04Check freshness
- 05Confirm action
Coordinates guide attention; they do not establish current truth.
Original conceptual diagram · not a live trace or measured result.Name the frame with every location
A coordinate without a frame is a vague instruction disguised as precision. Record whether a location is in image pixels, CSS pixels, viewport coordinates, document coordinates, normalized bounds, a 3D scene frame, or a physical-world frame. Include width, height, origin, orientation, capture time, device or camera identity where appropriate, and every crop, rotation, zoom, or transform applied after capture.
CSSOM View defines browser-facing concepts including viewport geometry, element layout positions, CSS pixels, scrolling, and visual viewport scaling. That does not make a screenshot coordinate stable. It shows why the interface must state which geometry it is using before translating “there” into an instruction.
Treat visual observations as time-bound
An image can support an observation about what it depicted when captured, not a guarantee about what is on screen now. Store capture or render time, source URL or screen identity, visibility scope, and a freshness policy. Before sending a pointer instruction or invoking a UI action, compare that evidence with the current layout or request a refreshed capture. If comparison is unavailable, use language that reflects uncertainty.
Hypothetical worked example: a user asks which invoice row to approve. The assistant identifies a row in a screenshot taken before a filter changed. Rather than click at coordinates, it says “I may be looking at an older view. Please confirm the invoice number shown in the current table,” then highlights the matching label if a new capture supports it.
Prefer semantic anchors to pixels
Where possible, ground a visual target in a stable name, accessible label, document identifier, field value, or object ID. Pixels are useful for drawing a highlight, but an operation should use the application’s semantic control where it exists. A button’s visible position may change with responsive layout, translation, zoom, or an open keyboard; its accessible name or controlled ID is often more durable.
This is not a recommendation to scrape hidden interfaces or bypass user control. The system should describe what it matched, show the target, and wait for confirmation for consequential steps. If no semantic anchor exists, treat a coordinate instruction as assistive guidance rather than an executable command.
Expose uncertainty in the interaction
Visual models can misread text, colors, occlusion, depth, and relationships. Avoid invented confidence percentages unless the system has a defined, calibrated meaning. Instead, show the source thumbnail or overlay, name the assumption, and offer concise correction choices: “Is this the red valve?” “Do you mean the upper-left card?” “The label is too small to verify.” A user correction should replace the active target, not append another competing interpretation.
For embodied systems, distinguish the avatar’s screen-space pose from a physical object location. An animated point or gaze can communicate attention, but it is not evidence of spatial measurement. Do not make a character appear to have seen a room or a device state that its actual sensor input did not provide.
Test transforms and stale views
Create fixtures for scroll, page zoom, pinch zoom, orientation change, responsive reflow, image crop and rotation, overlapping windows, camera movement, delayed frame arrival, and a target that disappears between recognition and action. Test paired examples where a blue control moves but its accessible name stays the same. Assert that stale or mismatched frames lead to a refresh or confirmation, never a blind click.
Success criteria: every visual target retains a frame and time; every transform is represented or invalidates the target; consequential action requires a current semantic target or explicit confirmation; and the user can see what the system is referring to. These are safety and usability gates, not claims about visual-model accuracy.
Leave a human escape hatch
Some visual tasks are inherently ambiguous: damaged equipment, small handwriting, a blocked camera, or a safety-critical physical layout. Give the user a way to describe the target in their own words, upload a clearer view under the file policy, or reach a person. A refusal to guess is a useful answer when the wrong target could cause loss or harm.
The limitation is fundamental: coordinate systems can make location representation precise; they cannot make an unclear image truthful. W3C specifications establish web geometry and pointer-event concepts, not a universal contract for visual interpretation or robotic action.
Take it into the review
Visual target record
| Field | Example | Review question |
|---|---|---|
| frame | CSS viewport, 1440×900 | what origin and units? |
| capture | render at current view | is it still fresh? |
| anchor | Approve invoice INV-18 | can it survive reflow? |
| transform | pinch zoom 1.5 | was target remapped? |
| action | highlight only | does user confirm? |
A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.
Primary reading
Sources and limits
These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.
What the sources establish
CSSOM View Module
The working draft defines web APIs for visual document geometry, including layout positions, CSS pixels, viewports, scrolling, and visual-viewport scale.
Limits: It is a working draft and does not define image-model grounding or real-world coordinate accuracy.
Checked 2026-09-19 · W3C CSS Working Group · source publication date not established.
Open original source ↗Pointer Events Level 3
The specification defines pointer event data and interaction concepts for web input.
Limits: Pointer events do not prove that a visual target remains present, correct, or authorized.
Checked 2026-09-19 · W3C · source publication date not established.
Open original source ↗Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.