The playbook

A support inbox to answers you have actually graded

Before any reply is sent to a customer, the same pipeline runs over old tickets where the right answer is already known.

6 stages Two days to build the grader, then continuous Claude API · OpenAI API Complete

Every stage names what goes in, what to do, why it sits here and what comes out, and the chain says where it breaks.

benchr has not run these capabilities itself. Each stage points at a record describing what a provider documents, read on the date shown.

01
Split
Your work
You start with
The archive of resolved tickets, each with the reply that closed it
Why this stage
The record and the prompt are both designed from these tickets, so a half read during that work can no longer judge it.
Do this
Cut the resolved tickets into two halves and set one aside, to be run but never tuned against.
You end with
A tuning half and a held-back half
The move Hold back a set the agent never sees
02
Record
Get JSON that always matches your schema
You start with
One resolved ticket from the tuning half and the policy page that answers it
Why this stage
A field that is not in the schema is not in the answers, and the answers are all the grader gets.
Do this
Define the answer record: the reply, the cited internal source, and a confidence field the model must fill.
You end with
A schema every answer must satisfy
The move Schema first, prompt second
What goes wrong here
- It returns JSON most of the time, then wraps it in prose and breaks my parser
03
Dry run
Run the whole job overnight at half price
You start with
Both halves of the split, the policy documents, and the schema
Why this stage
Nobody is waiting on a ticket that has already closed - the condition the asynchronous path asks for, and one a live inbox never meets.
Do this
Run the pipeline over a few hundred resolved tickets asynchronously.
You end with
Candidate answers for cases with known outcomes
04
Grade
Let the model write and run real code mid-answer
You start with
The candidate answers and the resolutions that closed those tickets
Why this stage
A silent model upgrade invalidates the old number, so the grading has to be a program that runs again, not a session of reading answers by hand.
Do this
Score the candidates against the known resolutions and compute the failure distribution rather than an average.
You end with
A grading report by ticket type
What goes wrong here
- It scores brilliantly on the demo and falls over on my actual work
05
Cache
Stop paying twice for the same file
You start with
The prompt you have stopped editing, and the grading report from the last run
Why this stage
The technique record warns that putting stable content first can change behavior. Do it before the grading report exists and nothing measures what the reorder did.
Do this
Move the policy documents and examples into the cached prefix once the prompt stops changing.
You end with
The same pipeline at a fraction of the input cost
The move Cache the part that never changes
What goes wrong here
- The prototype cost pennies and the real thing costs a fortune
06
Re-check
Let the model write and run real code mid-answer
You start with
The cached prompt, the held-back half, and the stored grading report
Why this stage
The last graded number was measured before the prefix moved. The same record that carries the warning also says to re-run the held-back set after.
Do this
Send the held-back half through the cached prompt and score it with the same program.
You end with
Two reports on the same tickets, before and after the reorder

How this goes wrong

An average holds still across a prompt change while one ticket type fails every time it appears. Read the count beside each rate as well - a category holding a handful of tickets swings on one of them.
The held-back half stops being held back the first time its failures are read and the prompt is edited to match them. After that there are two tuning halves and one number that measures neither.
Caching goes on while the policy text is still being edited. Each edit that lands inside the cached prefix invalidates it, so the run keeps paying the write premium on a prefix nothing reuses.
The record comes back valid on a ticket the policy does not cover: a fluent reply, a filled confidence value, and a citation to a page that does not say what the reply says. The shape is guaranteed; the content is not.
The prefix comes in under the cache minimum for the model being called. The usage block returns two zeros, no error is raised, and the requests have been paying the ordinary input rate the whole time.
The cost figure in the rehearsal is the batch rate. The pipeline that answers a waiting customer cannot use that path, so price it at the synchronous rate before anyone calls the rehearsal cheap.
Some requests come back with no usable record at all - a refusal, a safety rejection, a reply cut off at the token limit. Count those as failures of the run; dropped from the denominator they make the rate kinder than the pipeline is.
The grader reads real customer correspondence. Anthropic's code execution is not eligible for zero data retention, and it is unavailable on Amazon Bedrock and Google Cloud - settle that before the archive leaves your side, or grade on redacted copies.
The grading program sits under two documented ceilings: Gemini's sandbox caps runtime at 30 seconds, and skills running on the Claude API sandbox have no network access and cannot install packages at runtime. Keep each run small, with whatever the scoring needs already in the code.

A support inbox to answers you have actually graded

Split

Record

What goes wrong here

Dry run

Grade

What goes wrong here

Cache

What goes wrong here

Re-check

Where this leads

When it breaks

Technique

Capability

Goal