Where LLM Token Costs Come From—and How to Reduce ThemWhere LLM Token Costs Come From—and How to Reduce Them
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Chetan's avatar
Chetan Sreenivasa
• Oct 7
Where your LLM tokens actually go
The user's question is the smallest thing you send.
Most conversations about LLM cost start with the price list. Dollars per million tokens, this model against that one, which provider dropped prices last quarter. It's a reasonable place to start. It's also the wrong end of the problem.
Open your logs and pick one request. Not the bill, one request. Then count what's in it.
The question is a rounding error
Here's a typical chat request in a product with retrieval and a couple of tools. I've rounded the numbers, but the shape is what I'd expect to see:
The system prompt, with instructions, formatting rules and a few examples: around 1,200 tokens. The tool schemas, so the model knows what it can call: around 800. The chat history so far: around 2,400. The retrieved chunks from your vector store: around 3,000. And the thing the user actually typed: about 40.
That's roughly 7,440 tokens in, and the user wrote half a percent of them. Everything else is context your code decided to send. On every call. Whether it helped the answer or not.
Once you see a request laid out like that, the price list stops being the interesting part. A model that's 30% cheaper still bills you for all 7,400 tokens you didn't need to think about.
How a prompt gets fat
Nobody designs a 7,000-token request. It accumulates.
Someone adds an example to the system prompt to fix a bad answer. The example stays forever. Someone sets retrieval to top 10 "to be safe," because missing the right chunk felt worse than including nine wrong ones. Tool schemas get copied in for every call, including the ones that will never use a tool. And chat history just gets appended, turn after turn, because that's the easiest thing to write.
History is the sneaky one. Each turn resends every turn before it. A ten-turn conversation doesn't cost ten times one turn. It costs a lot more, because turn ten carries the weight of turns one to nine. The users who like your product most, the ones who keep talking to it, are the most expensive ones to serve.
None of these decisions was wrong when it was made. Together they're the bill.
What I'd do, in order
Measure by component, not by request. Log the token count of each part separately: system prompt, tools, history, retrieval, question, output. A single "tokens used" number tells you something is expensive. A breakdown tells you what. You can't cut what you can't see, and this is usually a day's work.
Bound the history. Keep the last few turns verbatim and summarise the rest, or drop it. Most conversations don't need turn two to answer turn eleven. Test it against your evals before you ship it, because some products genuinely do.
Retrieve fewer, better chunks. If the answer is usually in the top three, sending ten is paying for noise. A reranking step that trims ten candidates to three often pays for itself, and the model has less irrelevant text to get confused by.
Only send the tools the request can use. If you can tell from the route or the intent that no tool will be called, don't send the schemas.
Put the static part first and cache it. Several providers now discount input that repeats exactly across calls. That only works if your system prompt and tool schemas sit at the front, unchanged, with the variable parts after them. It's a small reordering with a real effect.
Then, and only then, look at a cheaper model. I'd make this choice last on purpose. Switching models with a bloated prompt just moves the waste to a cheaper invoice, and you lose the chance to see where it came from.
"Tokens get cheaper every few months"
They do. Prices have fallen fast, and I expect they'll keep falling.
But usage is growing faster in most products I hear about. Agents call the model several times per user action instead of once. Context windows got bigger, so people fill them. A falling unit price on a rising volume of waste is still a growing line on someone's budget.
And cost isn't the only thing a fat prompt hurts. Every extra token is latency the user waits through, and every irrelevant chunk is something the model might quote back as if it mattered. Trimming context tends to make answers faster and better at the same time. That's rare enough in engineering that I'd take it.
Cost is a design property
The bill shows up months after the decisions that caused it, and it usually lands with whoever owns the budget, not whoever wrote the prompt template. By then nobody remembers why retrieval is set to ten.
So I'd move the question earlier. When someone changes a prompt template, the review should ask what it costs per call, the same way a schema change gets asked about indexes. Not to block it. Just so someone knows.
What's the biggest slice in yours? My guess for most chat products is history. It's the one that grows quietly while everyone watches the retrieval settings.
AI Engineer OpenAILangChain
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started