benchr

When it breaks

Identical prompt, different provider, completely different behavior

Also asked as my prompt breaks on a different model · works on GPT, fails on Claude · the same instructions give different formats · porting between providers

Cause documented by a vendor or a published paper Not tested by benchr First logged Last checked

What is actually happening

The prompt is only part of the request. The features it silently relies on — schema enforcement, tool format, caching, system-prompt handling — differ per provider, and some are not available at all.

Why

  • Structured output, tool calling and prompt caching are each documented per provider, with their own support floors. A prompt that leans on one is not portable by copying the text.
  • Model identifiers are not interchangeable, and a retired identifier fails as a 404 rather than as a fallback.
  • Even where a feature exists on both, the documented limits differ — which parts of JSON Schema are supported, what counts as too large, how long a request may run.

The quick fix

Before blaming the prompt, check that the feature it depends on exists on the target model at all. Half of these failures are a missing parameter, not a wording problem.

The real fix

Separate the instruction from the mechanism. Keep the wording provider-neutral and keep the provider-specific parts — schema, tools, caching, model id — in configuration you can swap.

Step by step

  1. List every non-text feature the request uses: schema, tools, cache, file upload, system prompt.
  2. Check each against the target provider's current documentation, not against memory.
  3. Verify the model identifier on the provider's own model directory before switching.
  4. Re-run your held-out task set after the switch. Assume nothing carried over.
  5. Keep the differences in one adapter, so the next switch is a configuration change.

If you are seeing an API error

Grounded in

Where this leads

Technique

Capability

Goal

Workflow