Cache Invalidation Strategies in AI Retrieval Pipelines

Phil Karlton said there are only two hard problems in computer science: cache invalidation and naming things. He said it before anyone had built a production LLM pipeline, and the joke has aged badly, because AI retrieval systems didn't solve the problem. They multiplied it. A typical production stack carries at least four distinct cache layers running at once, and each one goes stale in a different way, on a different schedule, for reasons the other three layers can't see.

Why cache invalidation compounds in AI retrieval stacks

Traditional caching has a clean contract. You store a database row, a rendered HTML fragment, a computed price. When the underlying data changes, you invalidate the key, fetch again, and the cache is correct once more. Right or wrong is binary, and the machinery for keeping it right is decades old.

AI caches don't get that luxury. What they store isn't a fact, it's a response, and the correctness of that response depends on context, on how recently the source material changed, on which model version produced it, and on documents that might have been edited an hour ago by someone in another department. "Stale" in this world doesn't throw an error. It returns a confident, well-formatted, entirely wrong answer, and nothing in a standard monitoring dashboard flags it until a user notices the contract clause is the one from last quarter.

Layer that problem four times over (prompt cache, semantic cache, RAG retrieval cache, embedding index) and you get a system where each tier can be individually correct while the whole pipeline serves garbage. Fixing invalidation at one layer does nothing for the other three. That's the actual shape of the problem, and it's why a single invalidation strategy applied uniformly across a stack tends to fail quietly rather than loudly.

Agent Consumption of Cached Knowledge and the Importance of Invalidation Under Repeated Retrieval

Agentic systems generally run on several interlocking layers, a reasoning layer, an orchestration layer, a memory and data layer, and a tool integration layer. The memory and data layer is where caching lives. It holds short-term session context, pulls long-term knowledge across sessions, and gives the agent access to enterprise knowledge through RAG.

Agents don't ask a knowledge base a question once, which is what makes invalidation urgent rather than academic. Multi-hop reasoning means the same corner of a document store gets hit repeatedly within a single task, sometimes dozens of times, as the agent reformulates a query, checks a fact, backtracks, and checks it again from a different angle. That repeated-access pattern is the reason caching pays off. It's also what makes staleness expensive, because one bad cached answer doesn't get consumed once, it gets consumed by every downstream step that trusted it.

Consider the ordinary support-bot refund policy: "What's your refund policy?" Asked 47 times in a day, that question, without caching, regenerates an embedding and a full LLM response 47 separate times. That's the baseline case that makes semantic caching an obvious win, and it's also the baseline case that makes stale semantic caching an obvious liability once the refund policy actually changes.

The embedding index: the bottom-layer cache whose staleness is invisible

Every RAG system sits on top of an embedding index, the precomputed vector representations of every document chunk in the knowledge base. In theory the index rebuilds whenever a document changes. In practice, rebuilding is expensive, so it happens asynchronously, on a schedule, leaving a window where live retrieval runs against embeddings that no longer match the source text.

This layer fails in three distinct ways, and none of them look like a failure from the outside.

The first is semantic drift. A term's meaning in context shifts over time even when the document containing it doesn't change a single character. An embedding computed months ago correctly located that document in vector space at the time. It may no longer sit near the queries a user would write today, and nothing about the document itself would tell you that.

The second is model version mismatch, and this one catches teams by surprise more often than it should. Upgrading an embedding model is a cache-busting event, full stop, even though it rarely gets treated as one. Embeddings from two different model versions are not numerically comparable: a cosine similarity score of 0.9 under one model carries a different meaning than 0.9 under another. If the model version isn't part of the cache key, the system will keep serving similarity scores that look fine and mean nothing.

The third is the document-deletion gap. Batch pipelines remove a document from the source system, but the corresponding vectors can linger in the index for days, orphaned. A user gets pointed to content that no longer exists anywhere except in the cache. Solving this requires explicit deletion tracking and a cleanup job that actually runs, beyond an additive sync process that only knows how to add.

The RAG retrieval cache: when correct source documents produce stale retrieved chunks

One layer up sits the retrieval cache, which stores the output of a vector similarity search: given this query embedding, return these chunks. The key here is the query. That single design choice is the root of the layer's core failure mode.

Source documents update. The retrieval cache has no visibility into that update, because it was never keyed on document identity. There is, structurally, no invalidation path from "the document changed" to "the cached entry for that document is now wrong."

A lawyer updates a contract, then searches for it, and gets served the cached version of the old contract. The retrieval cache was keyed on the query embedding, not on the document ID, so there was no route by which updating the document could reach into the cache and evict the stale entry. Standard key-based invalidation simply has nothing to grab onto.

TTLs get reached for as the fix, and they help, but only partially. A TTL tells you when an entry was written. It tells you nothing about whether the source document has changed since. A long TTL on a retrieval cache entry means a compliance-critical policy update can sit unreflected in production for that entire window, served with total confidence, to anyone who asks.

The semantic cache: the highest hit rate, the most dangerous failure mode

Semantic caches operate one level above raw retrieval. Instead of matching on exact query strings, they match on embedding similarity, so a new query within a set cosine distance of a previously cached query gets that cached response, with no LLM call and no RAG pipeline touched.

"How do I return an item?" and "What's your return process?" sit above the typical similarity threshold, often set above 0.95 cosine similarity, so the second query rides on the first query's cached answer. It's an efficient mechanism, and the efficiency numbers back that up: semantic caching commonly delivers 40 to 70 percent hit rates in production workloads. An AWS-published evaluation running against 63,796 real chatbot queries found that, at the optimal similarity threshold, semantic caching cut costs by 86 percent and improved latency by 88 percent, while holding accuracy above 91 percent.

The latency gap alone explains why teams reach for this layer aggressively. A cache hit resolves in 2 to 5 milliseconds. A full LLM round-trip runs 300 to 500 milliseconds. That's a massive difference in order of magnitude, and users feel it immediately, no benchmark required.

The danger scales with the benefit. The exact property that makes semantic caching powerful (it treats similar-but-not-identical questions as the same question) is the property that makes it dangerous once the underlying answer changes. If the refund window shrinks from 30 days to 14, every phrasing of "what's your return policy" that previously clustered under the old cached answer keeps serving that old answer until something explicitly invalidates it. High hit rate and high blast radius are the same number, just described two different ways.

The prompt and KV cache: the provider-level tier with mechanical but unforgiving invalidation rules

Prompt caching, offered by providers including Anthropic and OpenAI, is a provider-side feature, not application logic. It's a provider-side feature that stores the computed key-value attention matrices for a stable prompt prefix, so the model doesn't have to recompute attention over that prefix on every single call.

The economics, at least for Anthropic's Claude, are specific. Cache-read tokens cost a small fraction of the base input token price, a substantial discount from the second request onward. Cache-write tokens cost a modest premium over the base price for a 5-minute TTL, or a larger premium over base for a 1-hour TTL. Latency drops by up to 85 percent on long prompts. For a system that prepends a 50,000-token system prompt to every request (a common pattern for agents carrying tool definitions and instructions), prompt caching turns what would be a large recurring cost into something close to negligible, once the cache is warm.

The invalidation rule at this layer is mechanical, and it does not bend. Any change to the cached prefix, even a single character, evicts everything downstream of that point. Move a frequently changing element higher up in a system prompt, intending to make updates cheaper, and the restructuring itself invalidates the entire cache for every user, all at once, until it rewarms from scratch. There's no partial credit here. The KV cache does exact-match on the prefix, not fuzzy matching the way a semantic cache does. It's exact-match on the prefix, or it's a miss.

Choosing the right consistency model for each tier

None of this argues for picking one invalidation strategy and applying it everywhere. It argues for the opposite: matching the consistency guarantee to what each tier actually needs, because over-engineering consistency at the wrong layer just burns latency without buying correctness.

Strong consistency, where every read reflects the most recent write, belongs on caches where being wrong has a clear downstream cost: financial data, compliance-critical policy documents, authentication decisions. It's rarely the right call for a semantic cache, because the coordination overhead required to guarantee strong consistency eats most of the latency advantage that made semantic caching worth building.

Eventual consistency fits semantic caches and most RAG retrieval caches reasonably well, provided two conditions hold: users can tolerate a brief window of slightly stale answers, and the underlying content changes far less often than it gets queried. Without a bound, "eventual" offers only a hope, not a guarantee. An entry that's eventually consistent but has no ceiling on how stale it can get is a liability wearing a technical-sounding label.

Bounded staleness sits between the two. For most production RAG retrieval caches, it is the right answer. The guarantee is concrete: no entry older than X seconds, full stop. A practical starting point looks like a differential TTL, tuned to how fast different content actually changes: a 7-day TTL for reference material that rarely moves, 1 hour for content that updates regularly, 5 minutes for anything tied to active operational decisions. None of that requires exotic infrastructure. It requires treating each cache tier as its own system, with its own definition of correct, rather than reaching for one invalidation rule and hoping it holds across all four layers at once. It won't hold across all four layers, because each tier has its own definition of correct.