benchrWhen it breaks
The browser agent clicks the wrong button
Also asked as browser agent clicks wrong thing · agent picked the wrong result · computer use clicks the wrong element · agent can't find the button · it selected the wrong row
Cause documented by a vendor or a published paper Not tested by benchr First logged Last checked
What is actually happening
Usually a perception failure wearing a navigation costume. The agent found the page; it misread which element on it was the one you meant.
Why
- A benchmark built around realistic cross-site work found visual reasoning over rendered content, rather than UI manipulation, to be the dominant failure mode.
- Two controls that look similar to a model are frequently not similar to a person: adjacent rows, repeated buttons, and anything identified by position rather than by label.
- Reading a value off a chart or a dense table is close to zero even for agents that navigate well, so any decision that depends on such a value is compromised before the click.
The quick fix
Refer to targets by their accessible name, not by where they are. 'The button labeled Continue' survives a layout change; 'the button on the right' does not.
The real fix
Give the agent structure instead of pixels. Where the site exposes an accessibility tree, an API or a feed, route the decision through that and leave the browser for the actions that genuinely need a browser.
Step by step
- Name targets by label or role, never by coordinates or ordinal position.
- Have the agent state what it is about to click and why, before it clicks, on any irreversible step.
- Never let an agent take a destructive or paid action without an explicit confirmation step you control.
- Where a value drives the decision, fetch it from the source rather than reading it off the rendering.
- Log the page state at each action. Without it, a wrong click is unreproducible and you will fix the wrong thing.
Grounded in
- On the hardest web tasks, people finish 10%. The best agent finishes 8%.The CAP preprint isolates perception, not manipulation, as the dominant failure mode, with near-zero performance reading values off charts.