benchrWhen it breaks
It described the chart correctly and then got the number wrong
Also asked as AI misreads charts · it invented a value from the graph · wrong figure from a table screenshot · it can see the dashboard but can't read it
Cause documented by a vendor or a published paper Not tested by benchr First logged Last checked
What is actually happening
Describing a chart and reading a precise value off it are different jobs. The first is a summary; the second requires locating a point against an axis.
Why
- A benchmark built specifically around this found visual reasoning over rendered content, rather than clicking and typing, to be the dominant failure mode for browser agents.
- Performance on extracting precise values from charts and tables was near zero in that evaluation, even for agents that navigated the sites competently.
- A confident description of the chart is not evidence that the value was read. It is evidence that the chart was recognized.
The quick fix
Give it the numbers, not a picture of the numbers. A CSV export, the underlying table, or the data endpoint removes the entire failure mode.
The real fix
Design the step so no value ever has to be read off a rendering: export the data, or have the model write code that reads the source, and reserve the image for context.
Step by step
- Ask where the underlying data lives before asking the model to look at a picture of it.
- If only an image exists, ask for the axis labels and gridline values first, then the point — and check them.
- Cross-check any extracted figure against a second source before it enters a document.
- Prefer 'write code that parses this file' over 'read this screenshot'.
- Treat any number that arrived through a screenshot as unverified until you have seen it in the source.
Grounded in
- On the hardest web tasks, people finish 10%. The best agent finishes 8%.The near-zero result on reading precise values from charts and tables is the CAP preprint's own finding.