Your token bill has a strange property: the price of tokens keeps falling and the bill keeps growing. Gartner's March 2026 analysis explains why — agentic models burn 5 to 30 times more tokens per task than a standard chatbot, and as consumption rises faster than unit costs fall, Gartner expects overall inference costs to increase. Cheaper tokens, bigger bills. The deflation is real and it will not save you.

Here's the part that should annoy you: a meaningful slice of that spend buys answers you already own. Research on query patterns has consistently found heavy repetition — the MeanCache paper cites prior studies showing roughly 33% of search-engine queries are resubmissions, and LLM traffic repeats the same way, just in more expensive clothing. "How do I reset my password?" and "steps to reset my password" are the same question. Your agent fleet asks both. You pay twice.

The obvious fix is a cache. The obvious cache is a trap. Let's do this properly.

Why exact-match caching barely helps

Traditional caching hashes the request and looks up the hash. It works when the same bytes arrive twice — which, for LLM prompts, they mostly don't. Natural language is a machine for producing near-duplicates that hash differently. An exact-match layer catches automated pipelines and user retries, and then goes quiet.

So the industry reached for semantic caching: embed the query, compare against stored queries by cosine similarity, return the cached response when similarity clears a threshold. Sound idea. The implementations are where the trouble starts.

Why naive semantic caching is a footgun

Two failure modes matter, and vendors are quiet about both.

The hit rates on the slide aren't the hit rates in production. One practitioner analysis puts it plainly: vendor decks tout "95% cache hit rates," but that figure describes match accuracy when a hit is found — real deployments see 20-45% of traffic actually served from cache. Still worth having. Not what the slide implied.

A false hit is worse than a miss. The same analysis documents a financial-services deployment where "I don't want this business account anymore" matched a cached payment-cancellation flow at 88.7% cosine similarity — when the correct behavior was an account-closure review. Set your threshold too loose and the cache stops saving money and starts manufacturing incidents. A wrong cached answer is a hallucination you engineered yourself, served instantly, at scale, with a straight face.

There's a third, quieter problem. Most semantic caches need an embedding model to do the similarity math — which usually means shipping every prompt to an embeddings API. The MeanCache authors flag exactly this: centralized caches accumulate everyone's queries, and embedding overhead can eat the savings the cache was supposed to deliver. To make LLM calls cheaper, you've added another external service that sees all your traffic. For a healthcare or legal tenant, that's not a footnote. That's a dealbreaker.

How Cortex does it instead

The semantic cache inside Cortex's LLM routing layer was built around those three failure modes. Every request through the router — including the OpenAI-compatible proxy at /v1/proxy/chat/completions — passes through a two-tier lookup:

Tier 1: exact match. The prompt is normalized and hashed with SHA-256, keyed to your tenant and agent. Sub-millisecond, zero false positives. This quietly catches the retry storms and pipeline duplicates first.

Tier 2: semantic match. On an exact miss, the prompt gets a TF-IDF embedding — computed locally, in-process. No embeddings API. No external vector database. Your prompts never leave the routing layer to be compared. The implementation is pure Python with a scikit-learn upgrade path, which means the similarity math costs microseconds, not another line item.

Around that lookup, the guardrails:

• The default threshold is 0.95 — deliberately conservative. We'd rather miss a valid paraphrase than serve the 88.7% wrong answer. You can loosen it per tenant when your traffic warrants it; the default assumes it doesn't.
• Entries are tenant-scoped with a TTL (one hour by default). Another customer's answers are never your cache hits, and stale answers age out instead of lingering.
• The cache fails open. Any error in lookup or storage logs a warning and falls through to a real LLM call. A broken cache costs you money, never correctness.
• It's off by default. Response caching changes behavior, so it's an explicit per-tenant opt-in (semantic_cache_enabled in your tenant's LLM config) — not a surprise we enabled on your behalf.

And because a cache you can't measure is a cache you can't trust: GET /v1/analytics/cache reports your hit rate and estimated savings over any window, alongside Cortex's cost analytics. You'll see what the cache is actually doing — in your traffic, not in a vendor benchmark.

The honest math

On FAQ-shaped and support-shaped traffic, semantic caching is at its best — the production data shows 40-60% hit rates there. On open-ended agentic calls, much less. If your workload mixes both, expect a blended rate in the 20s, not the 60s. At agentic token multipliers of 5-30x per task, a 25% hit rate on your repetitive traffic is real money — and unlike most cost levers, this one also makes those responses faster.

The cheapest LLM call is the one you don't make. The second-cheapest is knowing exactly how many you didn't make, and proving the ones you skipped were safe to skip.

Stop buying the same answer

Point your traffic at Cortex's LLM proxy — it's OpenAI-compatible, so it's a base-URL change — and check /v1/analytics/cache after a week of traffic. If your hit rate is near zero, congratulations: your workload is genuinely novel, and you've spent one GET request learning that. If it isn't, you've been paying full price for reruns.