TL;DR: In order of effort-to-impact: (1) turn on prompt caching (~90% discount on cached input at all three major vendors), (2) batch everything that isn’t latency-sensitive (flat 50% off everywhere), (3) route easy queries to budget models that are 25–50x cheaper than frontier, (4) stop stuffing raw data into the context window — it’s the biggest cost and the biggest accuracy killer. Pre-aggregating context before the model sees it is where the order-of-magnitude savings live.
Every company building AI on top of business data hits the same moment: the pilot was cheap, the rollout is not, and someone in finance is asking why the “AI line item” doubled again. Having spent the last two years building an AI-native platform — and paying these bills ourselves — here’s the honest map of what works, with sources and the caveats vendors leave out.
First, know what you’re paying (mid-2026 snapshot)
Two structural facts matter more than any individual price. Output tokens cost 5–6x input tokens at every major vendor — so verbose responses are a cost center all by themselves. And the gap between frontier and budget tiers is enormous: Anthropic’s Haiku 4.5 runs $1/$5 per million tokens (input/output) against $10/$50 for its frontier model; OpenAI’s gpt-5.4-nano is $0.20/$1.25; Gemini 2.5 Flash-Lite is $0.10/$0.40. That’s a 25–50x spread for work a budget model often does fine. (Prices from the vendors’ official pages as of July 7, 2026 — they move; date-stamp your own cost models. Anthropic’s Sonnet 5 intro pricing, for instance, ends August 31, 2026.)
The techniques, ranked by effort-to-impact
1. Prompt caching — the free lunch. All three vendors now price cached input at roughly 10% of standard: Anthropic charges 0.1x for cache reads (writes cost 1.25x, paying for themselves after one hit), OpenAI’s GPT-5.x caching is automatic at 10% of input price, and Gemini discounts ~90% but adds an hourly storage fee — which means short-lived caches of huge contexts can cost more on Gemini Pro. If your system prompt and account context repeat across calls, caching alone can halve a bill.
2. Batch APIs — the other free lunch. A flat 50% discount at Anthropic, OpenAI, and Google for asynchronous processing. CRM enrichment, transcript summarization, ticket classification — anything that can wait minutes instead of seconds should never run at synchronous prices. Caching and batching stack.
3. Model routing — real savings, inflated marketing. The defensible math is the 25–50x tier spread above: send the easy 80–90% of queries to a cheap model and reserve frontier for the hard ones. Tools like OpenRouter and the open-source LiteLLM make this operational (budgets, fallbacks, per-team tracking). Be skeptical of headline claims, though: router vendors advertise up to 97% savings and 10x cost cuts, but independent researchers (COLM 2025) found at least one commercial router that didn’t appear to be routing meaningfully at all as of late 2024. Measure on your own traffic.
4. Semantic caching — great for FAQ-shaped traffic. Serving repeated questions from a similarity cache cut API calls by up to 68.8% in published testing on customer-service workloads. Weak for one-off analytical queries over unique records — which is most of what customer teams ask.
5. Prompt compression — niche but real. Microsoft’s LLMLingua demonstrated up to 20x compression with ~1.5-point benchmark loss. Engineering-heavy; most teams get further with #6.
6. Stop stuffing the context window. This is the big one, and it’s a quality argument as much as a cost one. The research record is remarkably consistent: models attend poorly to the middle of long contexts (“Lost in the Middle,” TACL 2024 — 20–30 point degradation on multi-document tasks); accuracy decays well before context limits (Chroma’s “Context Rot”, 18 models tested); and Databricks found most models stop improving beyond 16–32k tokens of retrieved context — more retrieval can actively hurt. Meanwhile two of the three vendors surcharge long context (OpenAI’s long-context tier is ~2x input; Gemini Pro doubles above 200k tokens). “Just use the big window” costs more and answers worse.
7. Pre-aggregate before the model. The endgame. Anthropic’s engineering team published the canonical example: an agent workflow that consumed 150,000 tokens when raw tool results flowed through context, versus 2,000 tokens when data was filtered and joined before the model saw it — a 98.7% reduction on that workflow. Cloudflare independently landed on the same pattern. The insight generalizes: a two-hour call transcript is ~50,000 tokens, and a typical multi-tool workflow bills it through context repeatedly; a pre-built summary that already knows what matters about the account is a few hundred. Don’t pay frontier prices for joins and dedup — that’s ETL, and ETL should run once, not per prompt.
What this looks like when it’s the architecture
Pre-aggregation is the design principle Noded is built on. The Customer Context Graph continuously weaves your customer data — calls, emails, tickets, CRM records — into one organized story per account, before any model gets involved. When Claude, ChatGPT, or Gemini asks about an account (through one MCP server, not ten), it reads a curated brief, not eighteen raw exports. Teams see up to 90% lower token costs and around 40% better response accuracy — and because those are exactly the kind of numbers this post tells you to distrust, we published the methodology.
The honest caveat: if your AI usage is a handful of chat sessions a day, none of this complexity pays for itself — turn on caching, use a budget model, and move on. This stack matters when AI is in the workflow path, running against customer data all day. That’s when the token bill becomes an architecture question.
FAQ
What’s the fastest single change to cut LLM costs?
Prompt caching plus batch processing — roughly 90% off repeated input and 50% off asynchronous work, at all three major vendors, with no quality tradeoff.
Do LLM routers really save 90%+?
The savings are real when easy traffic moves to models 25–50x cheaper. The headline percentages are marketing; independent evaluation found at least one router not meaningfully routing. Benchmark on your own traffic.
Is long context cheaper than RAG or curation?
Generally no — OpenAI and Google surcharge long context, and published research shows accuracy degrades well before window limits. Curated context wins on both axes.
How does Noded fit in?
Noded does the aggregation step as infrastructure: customer data is organized once in the Context Graph, so every downstream model call is small and precise — whichever LLM your team uses.