benchr

Something is not working

What goes wrong when you actually use this, why it happens, and what to do instead. Every record says what no fix solves.

Unless a record is marked otherwise, a vendor page or a published paper explains the cause.

For

Documents and files

It says it read my PDF, but it only answered about the first few pagesAsk it one question it can only answer from the end of the document — a figure from the last table, the final section heading. If it cannot, the file did not arrive whole.

Output you can use

It returns JSON most of the time, then wraps it in prose and breaks my parserStop asking for JSON and start supplying a schema. The API feature is called structured outputs and it is a request parameter, not a prompt trick.

It stopped doing what I asked

Images

Video

The product looks different in every shot of my adGenerate from a still of the actual product rather than from a description of it, for every single shot.
Widely reported; no vendor documents the cause
The video shimmers — textures crawl and the light flickersShorten the shot. Most shimmer becomes invisible below about two seconds, and a cut costs nothing.
Widely reported; no vendor documents the cause

Agents and automation

Cost

The prototype cost pennies and the real thing costs a fortuneFind the largest unchanging block in your prompt. If it is re-sent every turn and not cached, that single change is usually most of the bill.

Limits and interruptions

It stops mid-task and tells me I have hit a limitRead the error before retrying. A 429 with a retry hint wants a wait; a quota error wants a smaller job, because no amount of backoff refills a balance.

If you are seeing an API error

Seeing an actual API error code? Those are catalogued separately.