LLM Token Cost Optimization: The Definitive Guide to Cutting Spend 30-90%
Cheaper models keep arriving, yet AI bills keep climbing. This is the enterprise playbook for the one line item that touches every model you run: the token. You will learn where token cost comes from, why it is the spend to get in front of now, the eight proven levers that cut it, and the open-source tools that pull each one, so you can turn a runaway bill into a managed one.
What LLM token costs really are (input vs output economics)
Every large language model bills in tokens. A token is roughly four characters of text, about three-quarters of an English word, so a 1,000-word document runs near 1,300 tokens. Each API call is priced on two token streams: the input you send (your instructions, context, and history) and the output the model generates. Prices are quoted per one million tokens, and the two streams do not cost the same.
Put governance around how your team uses AI. The AI Acceptable Use Policy: a deploy-ready template that sets the rules for AI use.
Your purchase helps keep our hubs free to read.
The gap is large. Output tokens cost 3x to 5x more than input tokens on standard models, and as much as 8x on reasoning models that generate long internal chains before answering. As a concrete anchor, GPT-5.5-pro is priced at $30 per million input tokens against $180 per million output tokens, a 6x premium baked into a single model.
That asymmetry is not arbitrary pricing. It comes from two physically different phases of inference. The prefill phase reads your whole input at once. It is processed in parallel across the prompt, so it is compute-bound and fast. The decode phase writes the answer one token at a time, and each new token depends on the one before it, so it cannot be parallelized. Decode is memory-bandwidth-bound and slow. You pay more for output because generating it is the expensive half of the machine, and the same fact explains why long answers feel slow: sequential decode dominates the latency a user actually sees.
The second structural fact is the price spread across the market. From a budget model like DeepSeek V4 near $0.44 per million input tokens up to a frontier endpoint like GPT-5.5-pro at $30, there is roughly a 100x range. Two models can answer the same easy question at prices two orders of magnitude apart. That spread is the raw material for the single largest lever you will meet later: sending each request to the cheapest model that can do the job.
Think of a request as two meters running at once. The input meter is cheap and fast. The output meter is expensive and slow. Almost every optimization in this guide is a way to slow one meter, shorten the other, or skip the model entirely.
Why token spend is the biggest AI cost to get in front of now
There is a paradox at the center of AI budgeting. The price of a fixed unit of capability keeps falling fast, yet total bills keep rising. Andreessen Horowitz named the price side of this LLMflation: the cost of an equivalent-performance token drops about 10x per year. GPT-3-class capability fell from roughly $60 per million tokens in 2021 to about $0.06 by late 2024, near a 1,000x decline in three years. If prices collapse like that, why do finance teams keep getting surprised?
Because cheaper tokens invite far more of them. When a call costs a fraction of a cent, teams stop rationing. They add retrieval, longer context, multi-step tool use, and background agents. Consumption outruns the per-unit discount, and the bill goes up even as each token gets cheaper. The lever that fights this is not waiting for prices to fall. It is managing how many tokens you spend and where. To model the full ownership cost behind those tokens, not just the per-token rate, run the numbers through our open-source vs frontier TCO calculator.
The steepest part of the curve is agents. A single chat turn is small. An agent that plans, calls tools, reads results, and re-plans is not. Oracle reports roughly a 4x token multiplier for agentic workflows over plain chat, rising to 15x for multi-agent systems, and a single complex task consuming up to 60 million tokens. The hidden driver is history: a 20-turn conversation that re-sends a large system prompt every turn can inflate a session from a few hundred tokens to well over 10,000, and cost scales quadratically rather than linearly if nothing intervenes. This is the agentic tax, and it is why token discipline moved from nice-to-have to load-bearing.
The governance gap is just as stark. In the same survey, 86% of engineering budget holders are unsure which AI tools deliver the most benefit, and 71% still budget with a naive one-to-one assumption between input and output tokens, which structurally understates the output-heavy cost you just met. On the overrun side, over half of AI teams report costs exceeding forecast by 40% or more during scaling, IDC projects up to 30% underestimation of AI infrastructure costs by 2027 for the largest firms, and real bills land about 2.8x over the original forecast on average. Left unmanaged, that gap roughly doubles the original bill.
The token cost lifecycle
Cost is not created at the moment of billing. It accumulates across a lifecycle, and every stage is a place to intervene. Understanding the stages tells you which lever attaches where, so you optimize the cause rather than the symptom. Read the bill at the end and you only see the total. Read the lifecycle and you see the leaks.
The rest of this guide walks the levers that attach to these stages, then gives you a framework for choosing which to pull first and a numbered procedure to execute them in order.
The 8 LLM token cost optimization levers (ranked by effort vs impact)
These are the eight documented ways to reduce token cost, ranked by the balance of effort and impact. Two carry the most weight at enterprise scale: model routing is the largest single financial lever, and prompt caching is the biggest structural change for anything multi-turn or agentic. The lowest-effort win, output and prompt discipline, needs no tooling at all. For each lever you get the mechanism, the documented savings range, when to use it, and the open-source tool that pulls it first, with a respected paid option second. Treat every percentage as a workload-dependent range, not a promise.
Send each query to the cheapest, fastest model that can handle its complexity. Simple utility work (classification, formatting, extraction) goes to a low-cost or local model. Expensive frontier endpoints are reserved for multi-step reasoning and coding. This exploits the roughly 100x price spread you met earlier. Router patterns range from rule-based (under 1ms) to embedding-based (about 5ms), semantic classifiers (50 to 100ms), and model cascades that escalate only when confidence is low.
During prefill the model computes a set of key-value (KV) attention tensors, the KV-cache, from your prompt. Prompt caching stores that computed state. When a later request shares an exact, token-for-token prefix, it reuses the cached KV state and skips prefill for that segment. It is provider-managed and prefix-exact: a single changed character before the cache boundary breaks the hit, which is why the winning move is to put static instructions and tools first and dynamic user variables at the very end. Output stays byte-identical because only the compute is reused, not the answer.
The savings are real but conditional. Anthropic and OpenAI advertise up to a 90% discount on cached input tokens, and Google implicit caching on Gemini 2.5 gives 75% to 90%. In multi-step agent workflows the overall cost reduction lands at 59% to 70%, with an 79% to 85% cut in time-to-first-token for long prompts. See the caching lever applied end to end in our Gemini context caching cost guide.
Trim redundant language, filler, and low-value blocks before the prompt reaches the target model. Techniques run from rule-based stop-word removal to extractive compression (named-entity and keyphrase extraction), token-level perplexity pruning, instruction distillation, and recursive summarization by a smaller model. Microsoft's LLMLingua uses a small aligned model to score token perplexity and drop the least informative tokens, reaching up to 20x compression with little performance loss (a 2,400-token prompt reduced to 115) and holding up on math and logic at high ratios. This is the strongest lever for novel, non-repetitive queries where caching cannot help, and it flattens the quadratic agent-loop growth curve.
This is the lowest-effort lever and the fastest latency win, because sequential decode is what users wait on. Set explicit max_tokens boundaries, write strict length rules into the system prompt (for example, answer in 50 words), and request structured formats. It attacks the 3x to 8x output premium directly. One content platform cut average generation from 500 to 300 words for a 40% drop in compute per article. Another cut token usage 30%, moving API cost from $100K to $70K per month, about $360K per year, with prompt discipline alone.
Group non-real-time requests into a single pass. Cloud Batch APIs from OpenAI and Anthropic apply a flat 50% discount on both input and output in exchange for asynchronous processing within a window of up to 24 hours. Self-hosted engines use dynamic or continuous batching to spread fixed model-weight overhead across concurrent requests, raising GPU utilization from a typical sub-30% idle to 70% to 90% and delivering 2x to 3x throughput, at the cost of about 20% added latency for larger batches.
Where prompt caching needs an exact match, semantic caching matches by meaning. Incoming queries are turned into vector embeddings and compared against a store of previously answered queries. If the semantic distance is below a threshold (for example, cosine similarity above 0.95) the stored answer is served and the model is never called, saving 100% of that request. Production workloads with high repetition report 20% to 70% overall reduction. GPTCache benchmarks show 61% to 69% hit rates at over 97% accuracy on hits; Redis LangCache reports up to about 73% cost reduction in high-repetition traffic.
Retrieval systems that dump the top 20 raw chunks into the prompt pay for context bloat on every call. Replace it with a RAG funnel (filter, rerank, then compress before generation) or layered memory that pulls only the precise facts the active prompt needs. Semantic chunking and a fixed retrieval token budget keep the context tight. Done well this cuts per-request cost 30% to 50% and can let a smaller model, 5x to 10x smaller, hit comparable quality because it is no longer wading through noise.
This is the self-hosted infrastructure lever for long-context, high-throughput workloads (very large prompts that fill the model's context window, the maximum number of tokens it can consider at once). The KV-cache can consume 40% to 60% of GPU memory for long contexts. Quantizing it to INT8 or FP8 cuts VRAM requirements 2x to 4x, which enables larger batches (higher throughput) and cheaper GPU classes. PagedAttention removes memory fragmentation, and optimized runtimes add continuous batching and kernel fusion for 2x to 3x throughput over a vanilla setup. It is higher effort and only pays off once you own the serving stack.
The levers at a glance
Ranked by the documented savings each can deliver on its own. The bars show the upper end of the cited range; real results depend on your traffic.
Ranges are workload-dependent and drawn from cited cases and benchmarks. Compression percentages apply to the prompt segment, not always the whole bill. Levers overlap, so combined savings are multiplicative, not additive.
Estimate your own token savings
Put your own numbers in. Enter monthly request volume, average input and output tokens per request, and a price tier, then check the levers you plan to apply. The estimator models your input and output tokens as two separate streams and applies each lever to the stream it actually affects: caching and compression act on your input tokens, output limits act on your generated tokens, and routing lowers the per-token price on both. Within a stream, overlapping levers stack multiplicatively at the conservative low end of each documented range. Prices are dollars per million tokens; every field is editable.
Estimates only. Caching and compression act on your input tokens, output limits act on your generated tokens, and routing lowers the per-token price on both, so the estimator applies each lever to the stream it actually affects rather than to the blended total. Figures use the conservative low end of each lever's documented range and stack multiplicatively within a stream. Actual savings are ranges, not guarantees: they depend on your traffic mix, cache hit rate, and quality constraints. Reverify per-token prices at your provider before budgeting.
Monitoring, attribution and FinOps: measure before you cut
You cannot optimize what you cannot see, and the provider's invoice does not show enough. A monthly bill tells you the total. It does not tell you which feature, team, or customer produced it, which prompts run long, or where a cache is missing. Billing-level data is a rear-view mirror. FinOps for tokens needs the telemetry one layer up, at the request.
Start with per-request and per-token attribution. Every call should log input tokens, output tokens, the model used, latency, and an estimated cost, tagged with the team, feature, or virtual key that made it. That single record answers the two questions the survey data says most teams cannot: where is the money going, and is it producing value. It is also what turns the value-blindness gap (86% of budget owners unsure which tools pay off) into a chart you can act on.
Layer budgets and alerts on top. A durable pattern is a per-key budget with an 80% alert threshold: each team or application gets a spending ceiling, and an alert fires at 80% so someone intervenes before the overage, not after the invoice. This is how you catch a runaway agent loop the day it starts rather than the month it bills.
Finally, change the denominator. Most teams track cost per token, which rewards using fewer tokens even when the output gets worse. The metric that matches the business is cost per outcome: cost per resolved ticket, per generated article, per closed task. Optimizing cost per outcome keeps you from starving a workflow that earns its spend while still cutting the ones that do not. Measure first, cut second, and you cut the right things.
Cutting spend from the invoice alone is guesswork. Without per-request attribution you cannot tell a wasteful workload from a profitable one, and blanket cuts hit both. Instrument first.
A decision framework for LLM token cost optimization: which lever to pull first
You do not apply all eight levers at once. You apply the one that matches the shape of your workload. The rule of thumb: attack the biggest, cheapest win first, then the biggest structural win, then refine. Answer one question at a time and the order picks itself.
Cap max_tokens, write length rules into the system prompt, and strip prompt boilerplate. It needs no tooling, returns 30% to 40%, and compounds with everything else. Then come back and answer the next question.
Front-load the static prefix, move dynamic variables to the tail, and enable provider caching. Expect 59% to 70% in agent loops, but run the break-even math on write-premium tiers first.
Send simple tasks to a cheap model and reserve the frontier for hard ones. This is the largest single lever, 40% to 85%. Add an evaluation gate so quality cannot silently regress.
Match queries by meaning and serve stored answers for near-duplicates, saving 100% on each hit. Tune the similarity threshold and measure accuracy on hits, not just hit rate.
Replace top-20 chunk dumping with a filter-rerank-compress funnel, then apply prompt compression. Expect 30% to 50% per request, and often a smaller model can now do the job.
For the build-versus-buy question in the same spirit, our total cost of ownership calculator models self-hosting against API pricing, and the model selector helps pick a cheaper model that still clears your quality bar.
The LLM token cost optimization playbook, step by step
This is the order to execute, front to back. Each step names what to do and what good looks like when it is done, so you can verify progress rather than guess at it.
- Assess. Map every workload calling an LLM: model, purpose, and rough volume. What good looks like: a one-page inventory that names each caller and the model it hits.
- Baseline and measure. Turn on per-request logging of input tokens, output tokens, model, latency, and estimated cost before you change anything. What good looks like: a dashboard showing spend by team, feature, and model for a full week.
- Pick levers. Run the decision framework above against each high-spend workload and write down which lever applies to which. What good looks like: a short list ranked by expected savings against effort.
- Cap output first. Set max_tokens and length rules, and strip prompt boilerplate. What good looks like: average output tokens per request drops with no measurable quality loss.
- Implement caching. Reorder prompts so the static prefix is first and dynamic variables are last, then enable provider caching. What good looks like: a rising cache hit rate and a falling blended input rate, after you confirm you are above break-even.
- Add routing. Introduce a router that sends simple tasks to a cheaper model, guarded by an evaluation gate on a held-out set. What good looks like: a majority of traffic on the cheap model with quality within your tolerance band.
- Compress and trim. Apply prompt compression to long unique prompts and replace top-k chunk dumping with a filter-rerank-compress funnel. What good looks like: shorter prompts at the same answer quality on your evaluation set.
- Set budgets and alerts. Give each team or key a ceiling with an 80% alert threshold. What good looks like: an alert fires in a test before any real overage occurs.
- Attribute to outcomes. Tie spend to a business outcome and switch your headline metric to cost per outcome. What good looks like: a cost-per-outcome number per workload that leadership can read.
- Iterate. Re-baseline monthly, keep the evaluation gate in continuous integration, and revisit the framework when traffic shifts. What good looks like: a standing review where each below-target workload has a named next lever.
Output caps and caching come before routing on purpose. They are cheaper to ship and they shrink the bill the router then optimizes, so every later lever operates on already-lean traffic.
The toolkit: open-source tools first, respected paid services second
Open source is the default recommendation. The projects below cover every lever, self-host cleanly, and carry no per-token markup. The paid column is the if-you-would-rather-buy-it alternative: reach for it when you want managed hosting, enterprise governance, or compliance coverage, not because the capability is missing from open source. Only tools documented in the grounding are listed, and capability claims are living data to reverify.
| Job | Open-source first | Paid, if you would rather buy it |
|---|---|---|
| Routing and gateway | LiteLLM, Bifrost, Helicone, Portkey Gateway, TensorZero | OpenRouter, Cloudflare AI Gateway, Vercel AI Gateway, AWS Bedrock, Azure AI Foundry, Kong Konnect |
| Prompt compression | LLMLingua, Bifrost Code Mode, Kong AI Gateway plugins | Available as gateway features within the paid platforms above |
| Semantic caching | GPTCache, Bifrost dual-layer cache, Helicone, LiteLLM (Redis-backed) | Redis LangCache, Braintrust Gateway (encrypted caching) |
| Inference and batching | vLLM, TensorRT-LLM, Text Generation Inference | AWS Bedrock / SageMaker (LMI containers) |
| Token measurement | Helicone, LiteLLM (per-key spend logging) | Included in the enterprise observability tiers below |
| Cost observability | Helicone, LiteLLM, Portkey Gateway, Langfuse | Portkey Enterprise, Datadog / Grafana / Splunk via gateway metrics |
A practical starting stack for most teams: LiteLLM or Bifrost as the gateway (routing, caching, and per-key spend in one place), GPTCache for semantic caching on repetitive traffic, LLMLingua for compression on long unique prompts, and vLLM only if and when you self-host. That covers seven of the eight levers before any invoice arrives. Grok API users can see the routing and caching pattern applied in our Grok API cost optimization guide, and teams weighing a cheaper self-hosted model should read running DeepSeek V4 cost-effectively.
Some vendor benchmarks (throughput multipliers, overhead in microseconds, hit rates, provider counts) are vendor-reported. Treat them as directional and confirm against your own workload before you standardize on a tool.
Test your understanding
Five quick questions to check the load-bearing ideas. Pick an answer for each, then reveal the explanations.
Frequently asked questions
What is a token, and why do output tokens cost more than input tokens?
A token is roughly four characters of text, about three-quarters of an English word. Output tokens cost 3x to 5x more than input tokens on standard models, rising to as much as 8x on reasoning models, because of how the two phases run. Input (prefill) is processed in parallel and is compute-bound and fast. Output (decode) is generated one token at a time, is memory-bandwidth-bound, and is slow. That asymmetry is both a pricing driver and a latency driver.
How much can token cost optimization actually save?
It depends on the workload, so treat every figure as a documented range rather than a guarantee. Model routing has cut total bills 40 to 85 percent in cited cases. Prompt caching has delivered 59 to 70 percent reductions in multi-step agent workflows. Prompt and length discipline alone returns about 30 to 40 percent with no new tools. Because levers overlap, stack them multiplicatively, not additively, and measure a baseline first.
Which lever should I implement first?
Start with the cheapest wins that need no infrastructure: set output length caps and tighten prompts, which returns roughly 30 to 40 percent. If a large prompt prefix repeats across requests, add prompt caching next. If your traffic is a mix of simple and hard tasks, add model routing to send easy work to a cheaper model. Route by the shape of your traffic, not by hype.
Does prompt caching ever cost more than not caching?
Yes, on tiers that charge a cache write premium. On Anthropic's 1-hour tier, a 30 percent hit rate can reach about 143 percent of the uncached baseline, and even 50 percent is roughly 105 percent. The 5-minute tier breaks even near a 30 percent hit rate. On no-write-premium tiers such as OpenAI and Google implicit caching, caching is net positive at all hit rates. Always run the break-even math, not just the headline 90 percent discount.
Are open-source tools good enough, or do I need a paid gateway?
Open-source tools cover most needs and are the default recommendation here. LiteLLM, Bifrost, Helicone, and Portkey Gateway handle routing, caching, and spend tracking. GPTCache and vLLM cover semantic caching and high-throughput serving. Reach for a paid service when you need managed hosting, enterprise governance such as SSO and audit trails, or SOC 2 and HIPAA coverage. Buy the operational burden away, not the capability.
Do I need to self-host models to cut token costs?
No. The largest levers, routing and caching, work entirely against hosted APIs with no self-hosting. Self-hosting adds infrastructure levers such as continuous batching and KV-cache quantization that pay off for long-context, high-throughput workloads, but they are the last mile, not the starting point. Most teams capture the majority of savings without ever running a GPU.
Everything in this guide is free to read. Members get the do-it-this-week kit that turns the playbook into artifacts you can hand to a team:
- Implementation worksheets for each lever
- The estimator's save and export mode
- Copy-paste procedure packs
- Config and policy templates
- A printable FinOps runbook