benchrThe playbook
A pile of PDFs to a structured dataset
Hundreds of documents become one table with a page citation behind every value.
4 stages A day to design the schema, then overnight per batch Claude API · OpenAI API Complete
Every stage names what goes in, what to do, why it sits here and what comes out, and the chain says where it breaks.
benchr has not run these capabilities itself. Each stage points at a record describing what a provider documents, read on the date shown.
- You start with
- The pile of PDFs
- Why this stage
- A partial read does not announce itself, so the check belongs before the extraction, not after it.
- Do this
- Design the schema first and keep it flat. The schema is the specification for the whole run.
- You end with
- A JSON Schema with a not-stated value for every field
What goes wrong here
- You start with
- The document text and page images
- Why this stage
- A schema is a contract. Asking for JSON in words is a request.
- Do this
- Upload each document once by file reference and extract against the schema, requiring a page number per value.
- You end with
- One record per document, with provenance
What goes wrong here
- You start with
- The extracted rows
- Why this stage
- A correctly shaped object can still hold a number the model read off a chart, which is the one thing it is measurably bad at.
- Do this
- Run the backlog asynchronously instead of one request at a time.
- You end with
- The full dataset at half the token price
What goes wrong here
- You start with
- The extracted dataset and the source pages it came from
- Why this stage
- A schema settles the shape and says nothing about the values, so a row can be well-formed and wrong. This runs as code rather than as another model pass because a range check and a total are the kind of thing a program decides the same way twice.
- Do this
- Validate the dataset: range checks, totals, and a sample re-read against the source pages.
- You end with
- A validation report naming the rows to check by hand