benchrWhen it breaks
It scores brilliantly on the demo and falls over on my actual work
Also asked as benchmark scores don't match reality · the agent works in the demo only · great numbers, useless in production · it overfits to the eval
Cause documented by a vendor or a published paper Not tested by benchr First logged Last checked
What is actually happening
You are comparing a number measured on a fixed set against work that is not on that set. The gap is not dishonesty; it is the difference between the two.
Why
- Published agent benchmarks are built and then optimized against. A team that tunes in a loop against its own eval set names overfitting as the loop's natural tendency, and holds back a validation split specifically to detect it.
- Benchmarks omit the shapes of work that are hardest. One team's own post gives 'extract 1,000 products with subpages and compare them across platforms' as an example nothing currently measures.
- On a benchmark built around realistic cross-site work, the same agents that top easier leaderboards finish 2–8% of tasks.
The quick fix
Take ten tasks from your own week, write down what a correct answer looks like, and run those. That is your benchmark; everyone else's measures someone else's work.
The real fix
Keep a held-out set the agent is never tuned against, and re-run it on every change. The number that matters is the one on tasks it has not seen.
Step by step
- Write ten real tasks with a checkable definition of done.
- Split them: some you iterate against, some you never look at while tuning.
- Re-run the held-out half after every prompt, tool or model change.
- Record the failures, not just the score. The failure list is what tells you what to fix.
- Re-run the whole set when the model version changes — a silent upgrade invalidates the old number.
Grounded in
- On the hardest web tasks, people finish 10%. The best agent finishes 8%.The 2–8% figures come from the CAP benchmark preprint, August 2026.