benchrThe playbook
A cost audit to a cheaper route, without changing the answer
Find out where the money actually goes before switching models, because usually it is repeated context rather than the model's rate.
4 stages Measured on live traffic, checked by hand, then priced Claude API Complete
Every stage names what goes in, what to do, why it sits here and what comes out, and the chain says where it breaks.
benchr has not run these capabilities itself. Each stage points at a record describing what a provider documents, read on the date shown.
- You start with
- A week of production responses with their usage blocks kept
- Why this stage
- Almost always the bill is one of three things: the same context re-sent and re-charged every turn, work that could wait running at the live rate, or an agent looping. This stage measures the first and the next moves the second; the third is a spend ceiling and an iteration cap, which the two fix pages linked from this chain carry and no stage here reaches. The published rates do not say which of the three you are paying for, so the reading comes before the route change.
- Do this
- Read cache_creation_input_tokens and cache_read_input_tokens on live traffic and find out how much of your input is unchanged between requests.
- You end with
- The share of your input that is repeated
What goes wrong here
- You start with
- The same week of traffic, with the place in the code that issued each request attached
- Why this stage
- Nothing in the usage block says who was on the other end, so the split comes from the call site, which is why this stage needs the code attached and the reading before it did not. The batch route is asynchronous by definition and anything user-facing cannot use it, so a request with a person waiting on it does not belong in the second group.
- Do this
- Separate the requests with a human waiting from the ones without, and move the second group asynchronous.
- You end with
- Two routes with different price points
What goes wrong here
- 03
Compare
Your work- You start with
- A held-out set of real requests from last week, and the prompt in the shape it had before the stable part moved to the front
- Why this stage
- The cache is a prefix, so making it hit moves the stable part of the prompt to the front, and the technique record's own note is that putting stable content first can change behavior and that the held-out set is re-run after. The batch route is not what is in question here; the ledger's expected line for it is the same output at half the token price. So the comparison is the prompt before the move against the prompt after it, run on a set frozen back at the measuring stage: picked now, it would be picked with the new answers already in view.
- Do this
- Run the set through the prompt as it was and the prompt as it is now, and read the two answers against each other yourself. A text diff tells you the wording moved; it does not tell you the answer got worse.
- You end with
- A pass, or the list of requests where the answer got worse
- You start with
- Last month's request and token counts, and the rates published today
- Why this stage
- A repeated share, a grouping and a set of answers that held are still not money. The arithmetic is the easy part; what decides the number is the published rate, and rates change - an old price sitting inside a cost model is a cost bug of its own.
- Do this
- Model the two routes against last month's real volume rather than an estimate.
- You end with
- A before-and-after number you can defend