Almost every AI pilot gets priced as though the pilot volume will be the production volume. That assumption creates the gap where a promising project turns into a difficult conversation with a finance director.
Inference cost, meaning the cost of asking a model to process and answer, has a shape. Teams often treat it as a simple line that rises with volume. The real bill usually behaves differently. Input costs dominate output costs, and the useful cost levers are often different from the ones teams reach for first.
This is the arithmetic we use on projects where the numbers have to stand up properly.
The pilot number usually understates the production bill
A pilot processes two hundred documents a month. The cost looks negligible. Everyone is delighted. The business case then uses the per-document figure from that run.
Production is forty thousand documents a month. Three things change at once, and only the first is obvious.
Volume rises two hundredfold, which everyone expects. The input mix also widens. The pilot used clean documents that someone selected. Production includes the scanned fax, the eighty-page appendix and the one in the wrong language. Those items use far more tokens per document than the pilot average. Tokens are the pieces of text the model reads and writes, and they are what the provider charges for.
Retries also appear. At pilot scale, transient failures are rare enough to disappear into the background. At production scale, they become a steady background rate. Each retry is another full-price call.
The realistic multiplier between a pilot per-unit cost and a production per-unit cost sits somewhere between one and a half and three, before any volume growth. Budgeting on the pilot number is the most common costing error in this field.
The bill is mostly driven by what you send to the model
On any retrieval or document system, the money goes overwhelmingly on input tokens.
People instinctively think about the answer being generated, because the answer is the visible part. In a typical retrieval query, the system sends several thousand tokens of context to produce two hundred tokens of answer. Retrieval is the part that finds relevant material and passes it to the model so the model can answer from it. Even where output is priced higher per token, the input side dominates because there is so much more of it.
That makes the highest-return cost lever simple to state. Send the model less material.
The best cost levers come before model choice
There are five main levers, and their order matters.
- Send less.
- Cache the repeated part.
- Route by difficulty.
- Fail early.
- Then consider the model.
Send less
Most systems send more context than they need. Retrieving ten chunks feels safer than retrieving four. When you measure accuracy at different k values, you frequently find that four performs identically to ten. That gives a sixty percent reduction in the dominant cost term with no quality loss.
The same pattern shows up in prompt boilerplate. A four-hundred-token system prompt repeated on every one of forty thousand monthly calls is sixteen million tokens a month of instructions the model has already been given.
Cache the repeated part
Where a provider supports caching a stable prefix, the saving is substantial and the engineering effort is low. The prompt needs to put the invariant part first and the variable part last.
That structure is a five-minute change in many systems. It often has never been made because the prompt grew organically.
Route by difficulty
Not every request needs the same model. A large share of production traffic is straightforward, and a smaller model handles it identically. Classify cheaply, route accordingly, and keep the expensive model for the cases that need it.
The trap is building the classifier with the expensive model, because that defeats the purpose. It usually needs to be a simple heuristic or a small dedicated model.
Fail early
Validate input before spending anything. A document that is out of scope, too large, in an unsupported language or unreadable should be rejected at the boundary. Letting the model discover the problem means paying full price for a case you already could have declined.
On one document system, adding a pre-check rejected eleven percent of submissions before any inference. That was eleven percent off the bill for a day of work.
Then consider the model
Teams usually start with model choice because it is a single configuration change. Once the four levers above have been pulled, it is usually the smallest lever.
The useful work is measurement. Run the evaluation set against the cheaper option and compare the result with the cost difference. We have moved production workloads to smaller models several times after finding no measurable difference on the task that mattered. We have also tried and reverted, which is why the measurement matters more than the instinct.
Better retrieval can beat a larger model
There is a point where a smaller model plus better retrieval beats a larger model plus mediocre retrieval, on both quality and cost. Finding that point is the most useful piece of analysis in most of these projects.
That point exists because retrieval quality and model capability can partly substitute for each other. A large model can compensate for being handed slightly wrong material by reasoning around it. A smaller one cannot. When retrieval is good enough that the correct passage is nearly always present, that compensation becomes unnecessary. The extra capability is still being paid for, even when the task does not use it.
On the engagement described in our piece on retrieval, fixing chunking and adding hybrid search took recall from 58 to 93 percent. At that point the smaller model matched the larger one on the evaluation set, at roughly a fifth of the per-call cost. The retrieval work paid for itself in about six weeks of production volume.
The bill includes more than inference
Focusing only on inference can understate the total, sometimes badly.
Several costs sit outside the per-token model call.
- Embedding the corpus.
- Vector storage.
- Compute to query vector storage.
- Evaluation runs.
- The human queue.
Embedding a corpus is a real one-off cost, and it becomes a recurring cost as documents change. A large corpus re-embedded because someone changed the chunking strategy is a surprise expense that has caught more than one team.
Vector storage and the compute to query it scale with corpus size rather than with request volume. That means they behave differently from everything else on the bill.
Evaluation runs also cost money. A scheduled daily evaluation over a set of eight hundred cases is eight hundred inference calls a day. Those calls produce no customer value and are absolutely worth paying for.
The human queue is a cost too. Any confidence threshold implies a volume of escalation, and that volume has a salary attached. A system with a ninety-five percent automation rate on forty thousand items still sends two thousand to people.
Model the cost before you build
The exercise takes an afternoon and prevents most of the unpleasant surprises.
Work through the model in this order.
- Estimate production volume, then double it, because adoption of something that works is faster than planned.
- Estimate average input tokens from real documents rather than clean ones.
- Separately estimate the ninetieth percentile, because the tail drives cost more than the mean.
- Add a retry factor.
- Add the embedding and storage lines.
- Add the escalation volume times a loaded hourly rate.
Then compute cost per business unit, whatever that is for the client. It might be per invoice, per claim or per ticket. Cost per token or per month is much harder for the business to evaluate. A finance director can immediately tell you whether eleven pence per invoice is good.
A worked costing shows where the money lands
Numbers make this concrete. This example is a document extraction system, with figures rounded and the sector changed, but the shape is real.
Volume
40,000 documents a month. Average input after retrieval and prompt: 3,200 tokens. Average output: 180 tokens. Ninetieth percentile input: 9,400 tokens, because a minority of documents are enormous.
The naive estimate
Someone takes the average, multiplies by volume, and produces a monthly figure. It is wrong by a wide margin in the same direction every time.
What actually lands on the bill
The tail matters. The top ten percent of documents consume roughly a third of all input tokens. Retries at a two percent transient failure rate add another two percent. Documents rejected after inference rather than before add whatever proportion of submissions were out of scope, which was eleven percent until we added the pre-check.
The non-inference lines
The non-inference lines in this system were:
- Initial embedding of the reference corpus, a one-off.
- Re-embedding when chunking changed, an unplanned repeat of that one-off.
- Vector storage, monthly and flat.
- Daily evaluation over 800 cases, which is 24,000 calls a month producing no customer output.
- 4 percent of documents escalating to a person, which at 40,000 volume is 1,600 human touches a month.
That last line was larger than the entire inference bill. On most systems we cost, the human queue is the dominant expense. It is also the one nobody puts in the model, because it does not arrive as an invoice from a provider.
The same estimating mistakes keep pushing costs upward
Four recurring errors all push the estimate in the same direction.
- Using the mean input size.
- Forgetting the system prompt.
- Ignoring the failure path.
- Excluding the humans.
Using the mean input size
Token cost is driven by the tail. Cost the ninetieth percentile separately and weight it, or you will be short.
Forgetting the system prompt
The system prompt is invisible in a notebook, and it is on every single call. Multiply its length by monthly volume before deciding it is not worth trimming.
Ignoring the failure path
Documents that cannot be processed still cost money if they reach the model. Rejecting them earlier is the cheapest optimisation available.
Excluding the humans
A confidence threshold is a budget decision disguised as a technical one. Raise it and quality goes up while the queue and its salary cost go up with it. That trade-off belongs in the model explicitly.
Provider price cuts do not remove the need for cost engineering
One reason teams underinvest in cost engineering is a belief that model prices fall fast enough to solve it. They have fallen, substantially, and that belief is still a mistake.
Prices per token have dropped while the tokens consumed per task have risen, because context windows grew and everyone filled them. Net cost per task has moved far less than the headline price cuts suggest, and on some workloads it has risen.
The non-inference lines do not fall at all. Storage, evaluation and the human queue are governed by your corpus, your quality bar and salaries. None of those track provider pricing.
The practical conclusion is to build the cost model against today’s prices, treat any future reduction as upside rather than as the plan, and spend the engineering effort on the levers you control. A system that is efficient because it sends less and rejects early stays efficient regardless of what providers do next.
The project is decided by cost per business unit
The number that decides everything is cost per unit against the cost of the process being replaced. That single comparison determines whether a project is worth doing. It is startling how often nobody has computed it.
If manual handling costs four pounds per document and the system costs eleven pence, there is enormous headroom and the engineering can be relaxed about efficiency. If manual handling costs twelve pence, the project is marginal and every lever above matters.
We ask for that comparison before quoting, and it is occasionally the reason we tell a client not to build. A system that saves three pence per unit on eight thousand units a year is not worth anybody’s engineering budget, and it is better to say so at the start.
More on how we price the work itself on engagement models, and on where AI genuinely earns its cost in AI development.
