Hold back a set the agent never sees

What the move is

Your own tasks, split in two. You iterate against one half; the other half is only ever run, never tuned against. The second number is the real one.

Why it works

A team that tuned an agent in a loop against its own evaluation set named overfitting as the loop's natural tendency and used a train/validation split specifically to catch it. The same tendency exists whether the tuner is an agent or a person.

Step by step

1. Write ten tasks you actually did this month, with a checkable definition of done.
2. Split them into a working set and a held-out set.
3. Iterate against the working set as much as you like.
4. Run the held-out set after every prompt, tool or model change, and record failures, not just the score.
5. Re-run everything when the model version changes — a silent upgrade invalidates the old number.

Paste this

<paste the held-out tasks, each with its own definition of done>

Run each task and report two things about it: whether it met its stated definition of done, and the first point where the output diverged. Do not propose prompt changes, do not rank the failures, and do not tell me how to fix them. This run is a measurement; advice turns it into another round of tuning.

Make it yours

Keep the held-out set small enough that running it is cheap. Five tasks you never tune against beat fifty you look at.

Variations

• Add one task you expect it to fail, as a check that the set can still discriminate.
• Have a colleague write the held-out set so your own assumptions are not in it.

Works with

• Hand off a task and walk away from the machine