Why the cost of AI catches people out

The economics of an AI project split into two very different halves: one-off investment and recurring cost. The surprises come almost always from the second.

Investment and recurring cost

Investments are data centres, chips and servers, training the model, acquiring data, and product development. They are largely paid up front.

Recurring costs are inference, electricity and cooling, cloud services, storage and data transfer, staff, and monitoring, security and customer support. They run for as long as the service is in use.

For a model company, training is an enormous outlay but not one-off in the way a building is: models go out of date, so training, post-training and evaluation repeat with every generation. For an application company, training is usually not a cost line at all: its bill comes from inference, which grows directly with use.

What sets the price of inference

The cost of a single answer depends on:

• the size of the model
• the length of the input
• the length of the answer
• how many reasoning rounds it takes
• the numerical precision used
• how many users are served at once
• how well queries can be handled in batches
• how busy the hardware is kept.

Most of these can be influenced by design. The choice of model is one variable among many.

Why the demo is cheap and production is not

A demo has one user, short inputs and one model call per answer. Production changes all of that:

• there are many users
• every input carries a lot of documents
• the answers are long
• an agent makes several model calls for one task
• the system uses the largest model even for easy work
• more capacity is reserved than is actually used.

The multiplication is the thing to understand. If one user request triggers ten model calls and each carries ten documents, the number of tokens read from documents goes up a hundredfold. The bill does not rise in quite the same proportion, because answer length, caching of the repeated part and whether the same model is used at every step all come into it. The order of magnitude is still that, and that is exactly what catches people out.

How the cost is kept down

The most common and most effective measures:

• The right model for the job. Classification, extraction and formatting often succeed on a small model. The large one is reserved for the hard cases.
• A shorter context. Retrieve only the relevant passages into the prompt, not the whole document.
• Caching. Store repeated questions and reusable parts.
• Batching. Run work that is not urgent when there is spare capacity.
• A ceiling on rounds. Give an agent a maximum number of steps and a maximum cost per task.
• Measurement. Cost per case, per user and per feature visible immediately, not first seen on the invoice.

Remember the other side of pricing

Alongside the cost, it is worth looking at what the customer pays for. If the pricing is a flat monthly fee but the cost arises from use, the heaviest users can turn the margin negative. Pricing tied to usage, or to the result achieved, keeps the two moving in step.