Semantic caching pitfalls: when cached answers go stale

Semantic caching is one of the highest-leverage cost optimizations available in an LLM gateway. Instead of sending a near-duplicate prompt to the model and paying for another inference, the gateway identifies that the incoming request is semantically equivalent to a previous request, and returns the cached completion. At meaningful traffic volumes, this produces 20–50% reductions in model spend with no degradation in user experience — when it works.

The failure mode is subtle and worse than a cache miss: returning an answer that was correct when it was generated but is now wrong. A user asking “what is our current refund policy?” gets the answer from three months ago. A developer asking “is this API endpoint still available?” gets a yes from before the deprecation. The user sees a confident, well-formatted response that happens to be false.

Understanding when semantic caching fails — and how to configure it to minimize those failures — matters more than knowing it exists.

How semantic caching works

A semantic cache stores a vector embedding of each prompt alongside the generated completion. When a new prompt arrives, the gateway embeds it and performs a nearest-neighbor search against the cache store. If the similarity score between the incoming prompt and a cached prompt exceeds a configured threshold, the cached completion is returned without calling the model.

The critical variables are:

• Similarity threshold — how close does the new prompt need to be to a cached one?
• TTL (time-to-live) — how long does a cached entry remain valid?
• Cache key scope — is the cache global, per-team, or per-user?

Getting any of these wrong produces incorrect behavior. Most deployments misconfigure at least one of them.

Pitfall 1: TTL that ignores content volatility

The most common mistake is applying a uniform TTL to all cached entries regardless of what the prompt is asking about.

A prompt like “explain the difference between a transformer and an LSTM” has a correct answer that doesn’t change. You could cache that entry for months. A prompt like “what is the current status of our deployment pipeline?” has a correct answer that might change every few minutes. Caching it for 24 hours guarantees you’ll eventually serve a stale response.

The fix is content-aware TTL policies. In ManyLayers Gateway’s semantic cache configuration, you can define TTL rules based on prompt metadata or detected content patterns:

semantic_cache:
  enabled: true
  default_ttl_seconds: 3600
  ttl_rules:
    - pattern: "current|today|now|status|latest"
      ttl_seconds: 60
    - pattern: "policy|pricing|tier|plan"
      ttl_seconds: 1800
    - pattern: "explain|what is|how does|define"
      ttl_seconds: 86400

Patterns are evaluated against the prompt text in order. The first matching rule wins. This doesn’t perfectly classify every prompt, but it substantially reduces the incidence of stale responses for time-sensitive queries.

Pitfall 2: similarity threshold set too aggressively

A high similarity threshold (closer to 1.0) means only very close matches hit the cache — you catch fewer savings but rarely serve the wrong answer. A low threshold (say, 0.80) means more cache hits but more false positives where semantically similar but meaningfully different prompts get the same cached response.

The failure case is instructive: “What are our enterprise pricing tiers?” and “What are our startup pricing tiers?” might score 0.87 on a cosine similarity metric — close enough to hit the cache at threshold 0.85 — but they are asking about different things and may have very different correct answers.

Tuning the threshold requires looking at false positive rates in your actual traffic. ManyLayers Gateway logs the similarity score alongside every cache hit. Run a sample of cache hits through a manual review: pull 100 random cache hits from the past week and check whether the served completion was actually a correct answer for the incoming prompt. If more than 5% are incorrect, your threshold is too low.

A starting threshold of 0.92–0.95 is more conservative than most defaults and produces fewer incorrect cache hits at the cost of a lower cache hit rate. Start conservative and move down only when you have data that supports it.

Pitfall 3: caching across contexts where answers differ

Some questions have different correct answers for different users or teams. “What models am I allowed to use?” has a correct answer that depends on the team asking. “What is the data retention policy for this workspace?” depends on the workspace. If the cache is global, the first team’s answer gets served to the second team.

Set the cache scope to match the scope of your content:

semantic_cache:
  scope: team   # options: global, team, user

For most enterprise deployments, team scope is the right default. It gives you cache benefits within a team’s usage patterns while preventing cross-team contamination. User-scoped caching is rarely worth the storage cost unless users have meaningfully personalized system contexts.

Additionally, when your application injects context into the system prompt (the user’s role, their permissions, their team’s configuration), those system prompt variations should be included in the cache key. ManyLayers Gateway includes the system prompt hash in the embedding by default — but if you’re stripping or normalizing system prompts before they reach the gateway, you may be unintentionally merging cache entries that should be separate.

Pitfall 4: caching responses that depend on retrieved content

RAG pipelines are particularly vulnerable to stale caching. If your application retrieves documents from a knowledge base and includes them in the prompt before sending to the model, the “same” user question might produce different completions on different days because the retrieved documents changed.

If retrieved content is included in the prompt, the semantic similarity of two prompts doesn’t capture the freshness of the underlying documents. A question like “summarize the key points from the product roadmap” might score 0.99 similarity against a cached prompt from last week — but if the roadmap document was updated yesterday, the cached summary is outdated.

The correct handling depends on your use case:

• Disable caching for RAG-augmented prompts entirely. Set a metadata flag on any prompt that includes retrieved content, and configure the cache to bypass on that flag.
• Cache with very short TTLs on RAG prompts. A 5-minute TTL on RAG-augmented prompts handles burst traffic (multiple users asking the same question in quick succession) without risk of serving stale content after an index update.
• Cache at the document summary level, not the prompt level. A pattern where the gateway caches individual document summaries and assembles them at query time is more cache-friendly than caching end-to-end RAG responses.

Pitfall 5: no invalidation mechanism

Even well-configured TTLs don’t cover all staleness scenarios. When you update your product documentation, publish new pricing, deprecate an API endpoint, or change a policy, the cache may hold entries that were valid yesterday and are wrong today.

ManyLayers Gateway’s semantic cache supports manual invalidation via the API and via the dashboard. When you make a significant content change that affects likely-cached queries, trigger a targeted invalidation. You can invalidate by:

• Tag (if you’ve labeled entries at insertion time)
• Pattern match on cached prompt text
• Full cache flush (last resort; use TTLs to rebuild gradually)

Build invalidation into your content update workflows. When the team publishes a new version of the knowledge base, a post-publish step that invalidates relevant cache entries prevents stale responses during the TTL window.

Measuring staleness in production

Set up a sampling pipeline that periodically re-evaluates cached entries by sending the original prompt to the model and comparing the fresh completion to the cached one. If the freshly generated response differs significantly (measured by semantic distance or by a grading model), flag the cached entry for invalidation and log the staleness event.

This is an asynchronous operation — it doesn’t add latency to user requests. Run it on a random 1% sample of cache hits, which at reasonable traffic volumes gives you a continuous freshness signal without meaningful cost.

Takeaway

Semantic caching is a high-value optimization that introduces a correctness risk if deployed carelessly. The risk is not theoretical — serving confidently wrong answers undermines user trust faster than serving no answer at all. Content-aware TTLs, conservative similarity thresholds, appropriate cache scoping, and an active invalidation practice turn semantic caching into a safe and effective cost lever.