benchrThe playbook
An undocumented repository to a documented test suite
Turn code nobody understands into tests that state what it is supposed to do - which is documentation that fails when it lies.
5 stages Several unattended runs over a week Claude Code · Gemini API Complete
Every stage names what goes in, what to do, why it sits here and what comes out, and the chain says where it breaks.
benchr has not run these capabilities itself. Each stage points at a record describing what a provider documents, read on the date shown.
- You start with
- The whole repository in one file, its token count, and the questions you want answered, one per pass
- Why this stage
- Count the tokens before the first pass. The million-token window on the documented Gemini models is a ceiling on what can be sent, not a promise of recall across all of it: the ledger records the window size, and records separately that usable recall across the whole window is a different question, with no published threshold for any specific model. A repository that does not fit is mapped one subtree per pass, and you join the maps yourself.
- Do this
- Ask one question per pass to map the modules, the entry points and the untested surface.
- You end with
- A map of what exists and what is untested
- You start with
- One module from the map, and the test command exactly as it is run
- Why this stage
- This stage rests on two lines in the rules file: the test command exactly as it is run, written as a command and not described, and the list of files that must not be modified. Leave the second out and an agent asked for a green suite has two routes to green - write tests that pass, or edit the module until the tests it wrote pass - and nothing in the request tells them apart. The file states the boundary rather than enforcing it: the control is the branch and the gate, not the file.
- Do this
- Write characterisation tests that record current behavior before changing anything.
- You end with
- A green suite that locks in today's behavior
- 03
Break
Your work- You start with
- The green suite, and the module in it you would least like to be wrong about
- Why this stage
- The suite was written from the code, so it agrees with the code by construction and green is the only color it has ever been. Its position is not negotiable: a test shape that catches nothing is the shape you package at the next stage as the convention for later runs. It is also the stage nothing else in the chain can do for you - the map lists modules without ranking them, so which one you would least like to be wrong about is yours to pick, and whether a red test names the behavior that broke or only goes red is a judgement rather than an exit code.
- Do this
- Change the module's behavior by hand - invert a condition, drop a field, return the wrong unit - run the suite, note whether it fails and whether the failure names the behavior, then put the code back. Repeat on the parts you would be sorriest to get wrong.
- You end with
- A list of the changes the suite let through
- You start with
- The tests that did catch a change, and the naming, layout and fixtures they use
- Why this stage
- What goes inside the skill decides whether it holds. Anything deterministic - the test command, the fixture layout, the naming check - belongs in a bundled script rather than in prose: a script runs through bash, so only its output enters the context window, and a script does not drift the way a convention left in prose does.
- Do this
- Package the house testing conventions as a skill so every later run follows them.
- You end with
- A skill the whole team's agents load
- You start with
- The skill, the CLAUDE.md at the repository root, and the modules from the map that still have no tests
- Why this stage
- One scheduled instruction is not one request. It expands into read, plan, edit, run the tests, read the failure, edit again, and a limit counts those calls rather than your instruction - so it lands inside the loop rather than between modules, and takes whatever that run had not committed. Committing between modules is what decides whether an interruption costs one module or everything since the last commit.
- Do this
- Schedule module-by-module runs and review each branch on its own.
- You end with
- One reviewable branch per module
What goes wrong here