Prompt Caching Strategies That Cut Your LLM Costs in Half

LLM API costs scale with token volume. For applications handling thousands of requests per day, the bill adds up fast — especially when many requests share common prefixes, similar queries, or identical system prompts. Caching is the most underused lever for reducing both cost and latency.

The Caching Landscape

There are three levels where caching applies to LLM applications:

Each level serves different scenarios and they can be combined.

Level 1: Application-Level Exact Match

The simplest and highest-ROI cache. If the same prompt produces a deterministic output (temperature=0), cache the response keyed on the prompt hash.

When this works: Classification tasks, structured extraction with fixed schemas, FAQ-style questions, any deterministic pipeline step. In a document processing pipeline where the same document might be reprocessed (retries, re-runs), exact match caching eliminates redundant LLM calls entirely.

Cache key design matters. Include everything that affects the output: messages, model, temperature, system fingerprint. Exclude request-level metadata (timestamps, request IDs) that would make every key unique.

Hit rates in practice: For RAG applications with a stable document corpus, we see 15-30% exact match hit rates. For classification tasks on recurring data, 40-60%. For conversational interfaces with unique user queries, under 5% — exact match alone isn't enough.

Level 2: Semantic Similarity Caching

When exact match hit rates are low, semantic caching catches queries that are phrased differently but mean the same thing. "How do I reset my password?" and "password reset steps" should return the same cached response.

return None

The economics are compelling. For a 10K-token system prompt:

• Without caching: 10K input tokens per request at full price
• With caching: 10K tokens at 1.25x once, then 10K tokens at 0.1x for subsequent calls
• Break-even: 2 requests. After that, every request saves 90% on the cached prefix.

OpenAI's automatic caching works without explicit markup. Requests with the same prefix (starting from the first token) automatically benefit from cached KV states. The discount is 50% on cached input tokens. No configuration needed, but less transparent — you only see the savings in usage reports.

Structuring Prompts for Maximum Cache Hits

Provider prefix caching only works when the prefix is identical. This means prompt structure matters:

Move all static content to the beginning. Put variable content at the end. This maximizes the cacheable prefix length.

For RAG applications, if you have a set of "always included" documents (product overview, company policies), put them in the cached prefix. Put query-specific retrieved documents after the cache boundary.

Cache Invalidation for LLM Applications

LLM caches have unique invalidation concerns:

When to invalidate:

• Model version changes (new model = potentially different outputs)
• System prompt changes (any edit should bust the cache)
• Underlying data changes (if cached responses reference data that's been updated)
• Quality issues detected (a cached response found to be wrong should be evicted)

TTL strategy: Use different TTLs based on the stability of the underlying data:

• Static knowledge (product docs, policies): 24-72 hours
• Semi-dynamic data (dashboards, reports): 1-4 hours
• Real-time data (stock prices, live metrics): Don't cache, or cache for seconds

Cost Impact: Real Numbers

For an internal knowledge base chatbot handling 10,000 queries/day with Claude Sonnet:

The combination of prefix caching and application-level caching is multiplicative, not additive. Prefix caching reduces the cost of cache misses, while application-level caching reduces the number of LLM calls entirely.

Implementation Checklist

1. Start with exact match caching on deterministic calls (temperature=0). Measure hit rate after one week.
2. Structure prompts for prefix caching. Move static content to the front. Mark cache breakpoints if using Anthropic.
3. Add semantic caching if exact match hit rate is below 20% and you have query volume to justify the embedding computation overhead.
4. Monitor cache metrics: hit rate, latency reduction, cost savings, and — critically — cached response accuracy. A stale or wrong cached response is worse than an expensive correct one.
5. Set up invalidation triggers tied to your data update pipeline. When the source data changes, evict affected cache entries.

Caching isn't glamorous, but it's the difference between an LLM feature that costs $50/day and one that costs $500/day with identical user experience. Start with the simplest strategy, measure the impact, and add complexity only where the numbers justify it.