benchr

Workflows

Some work is not a prompt. These are chains across more than one capability and more than one surface, with the artefact each step produces and the place each one usually breaks.

These records describe what providers officially document. benchr has not run these capabilities itself, so nothing here is a test result. Each record shows the vendor page it was read from and the date.

Ledger updated: September 3, 2026

Screenshot to a reviewed pull request

A picture of a screen becomes a branch with the component built, the tests written and the diff ready for a human.

Realistic time: An hour of attention, mostly in reviewDocs checked September 1, 2026

  1. One screenshot in, a working page out

    Turn the screenshot into a standalone component with an explicit list of assumptions.

    Produces: One self-contained file plus assumption comments

  2. Work through an entire repository, not a pasted file

    Move it into the codebase, replace the invented tokens with the project's real ones, and wire it into the route it belongs to.

    Produces: A branch with the component in its real home

  3. Let the model write and run real code mid-answer

    Generate and run the tests, and read the failures rather than describing them.

    Produces: Passing tests, or a named failure

  4. Hand off a task and walk away from the machine

    Open the pull request and let the review run without you.

    Produces: A reviewable diff

Where it breaks

  • The first pass invents design tokens. Step two is not optional - the assumption comments are what makes it fast.
  • Tests written against generated markup test the generation, not the requirement. Write at least one test by hand.
  • A screenshot of a state (hover, error, empty) is not the same as a screenshot of the component. Capture the states you need.
Works on
Claude Code, Claude API
Tasks
Turn a screenshot into a working page

A pile of PDFs to a structured dataset

Hundreds of documents become one table with a page citation behind every value.

Realistic time: A day to design the schema, then overnight per batchDocs checked September 1, 2026

  1. Get JSON that always matches your schema

    Design the schema first and keep it flat. The schema is the specification for the whole run.

    Produces: A JSON Schema with a not-stated value for every field

  2. Read a 600-page PDF including its charts

    Upload each document once by file reference and extract against the schema, requiring a page number per value.

    Produces: One record per document, with provenance

  3. Run the whole job overnight at half price

    Run the backlog asynchronously instead of one request at a time.

    Produces: The full dataset at half the token price

  4. Let the model write and run real code mid-answer

    Validate the dataset: range checks, totals, and a sample re-read against the source pages.

    Produces: A validation report naming the rows to check by hand

Where it breaks

  • A schema with no explicit not-stated value forces the model to invent one. This is the single most common cause of a confidently wrong dataset.
  • Scanned pages inherit every vision limit - approximate localisation and unreliable counting - so a scanned batch needs a higher hand-check rate.
  • Both the 32 MB and page ceilings apply to the whole request, not just the file. Split before you discover this.
Works on
Claude API, OpenAI API
Tasks
Get clean data out of a stack of PDFs, Get output a program can consume, every single time

A site with no API to a daily feed you own

Drive the browser once to learn the path, then freeze that path into a connector that runs without a model in the loop.

Realistic time: Half a day once, then unattendedDocs checked September 1, 2026

  1. Drive a real browser, not a scraper

    Have the agent find the data by reading the accessibility tree, and record the exact click path and element references it used.

    Produces: A written, reproducible path to the data

  2. Get JSON that always matches your schema

    Define the record shape the feed will publish, before automating anything.

    Produces: The feed's schema

  3. Give a model your own tools and data through MCP

    Wrap the frozen path as a read-only server so every client can call it without driving a browser.

    Produces: A connector with credentials on the server side

  4. Hand off a task and walk away from the machine

    Schedule the run and alert on schema violations rather than on content changes.

    Produces: A daily feed and a failure alert

Where it breaks

  • Element references go stale as soon as the DOM changes materially. Alert on the schema, because that is what actually breaks.
  • Page content is a prompt-injection surface. Never let the page's text decide what the agent does next.
  • Running a browser agent daily when a connector would do is the expensive mistake here - the model is for discovery, not for the schedule.
Works on
Claude API, MCP servers, Claude Code
Tasks
Automate something on a site that has no API, Let a model use your systems, not just your prompts

An undocumented repository to a documented test suite

Turn code nobody understands into tests that state what it is supposed to do - which is documentation that fails when it lies.

Realistic time: Several unattended runs over a weekDocs checked September 1, 2026

  1. Fit a whole codebase in one prompt, and find its limit

    Ask one question per pass to map the modules, the entry points and the untested surface.

    Produces: A map of what exists and what is untested

  2. Work through an entire repository, not a pasted file

    Write characterisation tests that record current behaviour before changing anything.

    Produces: A green suite that locks in today's behaviour

  3. Teach the model a procedure once and reuse it

    Package the house testing conventions as a skill so every later run follows them.

    Produces: A skill the whole team's agents load

  4. Hand off a task and walk away from the machine

    Schedule module-by-module runs and review each branch on its own.

    Produces: One reviewable branch per module

Where it breaks

  • Characterisation tests lock in bugs as well as behaviour. Label them as such or you will defend the bug later.
  • One question per pass matters here: multi-fact questions over a long context are exactly where documented accuracy drops.
  • An unattended run with no CLAUDE.md invents conventions, and you will review style instead of logic.
Works on
Claude Code, Gemini API
Tasks
Understand a codebase nobody has explained to you, Hand off work and get a result, not a chat

Raw meeting audio to decisions and owners

An hour of recording becomes a short list of what was decided, who owns it, and the timestamp that proves it.

Realistic time: Minutes per meeting once it is wiredDocs checked September 1, 2026

  1. Transcribe a recording and label who said what

    Split under 25 MB, transcribe with speaker separation, and keep the timestamps.

    Produces: An attributed, timestamped transcript

  2. Get JSON that always matches your schema

    Extract decisions and actions against a schema that requires a speaker and a timestamp for each one.

    Produces: A structured decision list with provenance

  3. Run the whole job overnight at half price

    Backfill the archive of past recordings asynchronously.

    Produces: A searchable history at half price

Where it breaks

  • Chunk boundaries split sentences and speaker turns. Overlap the chunks or accept a lost line at every seam.
  • Speaker labels are identifiers, not identities. Map them to people once, by hand, per meeting.
  • An action item with no timestamp cannot be checked, and an unverifiable action item is worse than none.
Works on
OpenAI API, Claude API
Tasks
Turn a recording into something you can search and quote

A support inbox to answers you have actually graded

Before any reply is sent to a customer, the same pipeline runs over old tickets where the right answer is already known.

Realistic time: Two days to build the grader, then continuousDocs checked September 1, 2026

  1. Get JSON that always matches your schema

    Define the answer record: the reply, the cited internal source, and a confidence field the model must fill.

    Produces: A schema every answer must satisfy

  2. Run the whole job overnight at half price

    Run the pipeline over a few hundred resolved tickets asynchronously.

    Produces: Candidate answers for cases with known outcomes

  3. Let the model write and run real code mid-answer

    Score the candidates against the known resolutions and compute the failure distribution rather than an average.

    Produces: A grading report by ticket type

  4. Stop paying twice for the same file

    Move the policy documents and examples into the cached prefix once the prompt stops changing.

    Produces: The same pipeline at a fraction of the input cost

Where it breaks

  • An average score hides the category that fails every time. Always read the distribution.
  • Grading on tickets the prompt was written from is measuring the fit, not the model. Hold a set back.
  • Caching a prefix you are still editing wastes the write premium. Cache last, not first.
Works on
Claude API, OpenAI API
Tasks
Get output a program can consume, every single time, Cut the API bill without changing the answer

A cost audit to a cheaper route, without changing the answer

Find out where the money actually goes before switching models, because usually it is repeated context rather than the model's rate.

Realistic time: An afternoonDocs checked September 1, 2026

  1. Stop paying twice for the same file

    Read cache_creation_input_tokens and cache_read_input_tokens on live traffic and find out how much of your input is unchanged between requests.

    Produces: The share of your input that is repeated

  2. Run the whole job overnight at half price

    Separate the requests with a human waiting from the ones without, and move the second group asynchronous.

    Produces: Two routes with different price points

  3. Let the model write and run real code mid-answer

    Model the two routes against last month's real volume rather than an estimate.

    Produces: A before-and-after number you can defend

Where it breaks

  • Switching to a cheaper model before caching is the classic mistake - it changes answers to solve a billing problem.
  • Two zeros in the cache usage fields mean the prompt was under the minimum, and no error is raised to tell you.
  • A one-hour cache costs twice base input to write. It only pays off if the prefix is genuinely reused.
Works on
Claude API
Tasks
Cut the API bill without changing the answer

An idea to a narrated product clip

Stills first, motion second, voice last - the order that keeps control where you still have it.

Realistic time: A day, most of it iterating on stillsDocs checked September 1, 2026

  1. Generate an image with legible text inside it

    Generate the key frames with the exact copy quoted in the prompt and brand references attached.

    Produces: Approved stills with correct wording

  2. Generate video from a prompt or a still

    Animate each approved still as a separate short clip - one camera move, one action each.

    Produces: Short clips you can cut together

  3. Talk to it out loud and hear it answer

    Use the realtime voice stack for narration only if you need a live take; otherwise script it and record once.

    Produces: A narration track

Where it breaks

  • Generating motion before the stills are approved multiplies the iteration cost by the hardest step.
  • Every generated image carries a SynthID watermark - know that before it becomes a client deliverable.
  • The video page benchr checked publishes no length or resolution limits, so plan the edit around clips you have actually produced rather than a spec you assumed.
Works on
Gemini API, OpenAI API
Tasks
Generate visuals where the words are spelled right