Teams budget for model calls and then get a bill shaped by retries, context growth and review time. Those grow at different rates, and only one of them is fixed by choosing a cheaper model.
Four curves, not one
- Per-call cost — flat, predictable, and the one everybody models.
- Context cost — grows with conversation length and retrieved volume. Each call is billed linearly in tokens, but the total compounds: every turn resends the history, and a bigger window invites more retrieval to fill it.
- Retry cost — grows with failure rate and with how far into a run failures occur. A retry at step nine costs nine steps.
- Human cost — grows with how often output needs review. This is usually the largest number and rarely appears in the model budget at all.
Cache the boring half
Most production prompts are mostly stable: system instructions, tool definitions, retrieved policy text. Prompt caching turns that from a full per-call charge into a heavily discounted one, for as long as the cache entry lives. It requires putting the stable content first and keeping it byte-identical, which is a small architectural discipline with an unusually direct payoff.
Route by difficulty, not by default
Most requests in a mature system are easy. Classification, extraction and routing rarely need a frontier model; the hard tail does. Measure the split before assuming — teams are routinely surprised that eighty per cent of traffic is trivial. Route the easy majority to a small model, keep the large one for the tail, and check the boundary with evals rather than intuition.
Retrieve less, more precisely
Stuffing twenty passages into context costs money on every call and usually lowers quality. Reranking to five is cheaper and better: you pay once for a small reranker instead of repeatedly for tokens the model has to wade through.
Spend on reliability to save on review
Human review is the dominant cost in most deployed systems, and it is bought down by precision. A feature that is right ninety-nine per cent of the time with a calibrated refusal costs far less to operate than one that is right ninety-five per cent of the time and always confident — because the second needs every output checked.
What to take away
Instrument cost per successful outcome, not cost per call. A cheaper model that fails twice as often is not cheaper. The metric that matters is what it costs to get one piece of work correctly done.