I Dug Into Prompt Caching and Found That Hit Rate Isn’t the Goal#

It Started With “Where Does This Actually Happen?”#

I knew prompt caching reduced costs. My understanding went about as far as: when you send the same system prompt repeatedly, it gets cheaper from the second request onward.

But the more I thought about it, the stranger it seemed. If caching happens in some relay layer outside the model, then from the model’s point of view nothing is saved. And yet the savings are said to reach 90%. Would that mean most of the cost of using an LLM (Large Language Model) comes not from the model itself but from the layer around it?

When I dug in, it turned out my premise was wrong from the start. And along the way I ran into a more uncomfortable fact: raising the cache hit rate is not the same thing as optimizing cost. This is that record.

First, What Is the “KV State” That Gets Cached?#

Attention in a transformer produces three kinds of vectors for every token.

When generating a new token, its Q is compared against the K of every preceding token to score relevance, and those weights are used to mix the V values before passing the result to the next layer. So K and V are “the reference material that earlier tokens provide to later ones.”

The key point is that K and V are determined solely by the token itself and the tokens before it. No matter what comes after, an already-computed K · V pair does not change. Q, by contrast, is only ever needed for the newest token, so there is no reason to keep it.

So the stack of K and V tensors for every token at every layer — that is the KV cache. For a 100K-token system prompt, that pile of tensors runs to several gigabytes. Prompt caching means holding onto that pile instead of discarding it, and reattaching it on the next request.

This is also where the “change one byte and everything after it breaks” rule comes from. If the third token changes, its K · V changes; the fourth token is computed with reference to that changed third token, so it changes too, and the invalidation cascades to the end. Everything before it is untouched. The reason caching works on a prefix basis turns out to be the structure of causal attention itself.

Caching Happens Inside the Model, Not Outside It#

Now back to the original question. Splitting the candidates three ways makes it clear.

So a cache hit means “the model does not run the forward pass for that span at all.” It skips the entire chain of computation — embedding, per-layer QKV projection, attention, feedforward — reattaches the stored K · V tensors in memory, and computes only the newly arrived tail.

That is also why every caching rule turns out to be explained by conditions inside the model.

• Switching models breaks the cache — another model’s K · V tensors differ in both dimension and meaning, so reuse is physically impossible
• The minimum token count varies by model — layer counts and head configurations differ, so the break-even point for a single cache entry differs
• The first response must begin streaming before you can read it — the tensor write has to actually complete

So What Is Actually Saved?#

LLM inference splits into two stages.

Prefill compute is roughly 2 × parameter count × input token count. Feeding a 100K-token prompt again every time means redoing that much matrix math from scratch on every request. Caching eliminates it wholesale.

So what gets saved is precisely the model’s GPU compute — the exact opposite of my original assumption.

Then What Is the 0.1× on Cache Reads Paying For?#

If the compute is gone, why isn’t it free? Because some costs remain.

1. Storage occupancy — those multi-gigabyte tensors have to be held in a fast memory tier for the duration of the TTL (Time To Live). That space cannot be used to serve other requests
2. Load bandwidth — the I/O cost of pulling the cache back into compute memory
3. Residual compute — the newly arrived tail still has to run attention against the entire cached K

There is decisive evidence that the first of these dominates: write pricing differs by TTL.

The compute is identical in both cases. Same prompt, same prefill. The only difference is how long it is held, and yet the price differs by 0.75×. That gap is the storage occupancy cost — which, read the other way, means that of the 1.25×, about 1.0× is compute and 0.25× is five minutes of storage rent.

In short, token cost is roughly GPU compute + memory bandwidth + memory occupancy, and all of it is the cost of running the model itself. Prompt caching can cut 90% not because the outer layer is expensive, but because prefill compute was that large to begin with and was pure duplication.

One Rule Only: Prefix Matching#

Once you understand this much, every caching rule compresses into a single sentence.

Caching is prefix matching. Change one byte anywhere in the prefix and everything after that point is invalidated.

And the physical order in which a request is assembled into a prompt is fixed.

tools  →  system  →  messages

tools sits at position 0. Add one tool or reorder them, and the cache for the system prompt and the entire conversation history is wiped. Conversely, placing a cache boundary on the last system block caches tools and system together.

What “Setting a Breakpoint” Actually Means#

To settle the terminology first: marker, breakpoint, and cache boundary all mean the same thing, and the concrete artifact is a cache_control field attached to a specific block in the request JSON.

This is where the biggest misconception arises.

• Wrong reading: “cache this block” (store just the one block)
• Right reading: “store everything from the start of the prompt through the end of this block as one cache entry”

It is a boundary marker, and the entire accumulation up to that point is what gets stored. Suppose a prompt made of six blocks has cache_control attached to the fourth.

It is not that the fourth block alone is stored. Blocks 1 through 4 are bundled together into a single cache entry. And if blocks 1 through 4 are byte-for-byte identical on the next request, that whole span is reused at 0.1× and only blocks 5 and 6 are computed fresh.

Let me compare using a case where you ask three questions about the same document. Putting the marker at the very end of the prompt goes like this.

Request 1:  [doc][question A ★]  → stores "doc + question A" (pays 1.25×)
Request 2:  [doc][question B ★]  → prefix differs → miss, stores again
Request 3:  [doc][question C ★]  → miss again, stores again

Result: all three cost more than list price, with zero reads. A net loss.

Moving the marker to the end of the shared span changes things.

Request 1:  [doc ★][question A]  → stores through "doc" (1.25×)
Request 2:  [doc ★][question B]  → identical through ★ → hit (0.1×)
Request 3:  [doc ★][question C]  → hit (0.1×)

The prompt content is exactly the same in both cases. The only difference is whether the marker went after the document or after the question.

As a one-sentence rule: put the marker at the last point where you can be confident “everything up to here will be identical on the next request too.”

Worth noting that this caching is opt-in. If you do not include cache_control, no caching happens no matter how large a system prompt you resend. There is also an “automatic caching” mode where you put cache_control at the top level of the request — but that is automatic placement, not automatic activation. You still have to include the field.

So You Design the Prompt as Strata#

Caching optimization turned out to be layout design, not a technique for placing markers well. Treat the prompt as sediment in layers of differing change frequency, and stack it so that the further forward, the harder the rock.

Every practical guideline derives from this picture. Here are a few, ordered by impact.

1. Treat the system prompt as a build-time constant. Do not assemble it with string interpolation at runtime. The moment you put the current time, a user name, or a session ID into the system prompt, your hit rate is permanently zero. This single issue was the most common thing separating 0% from 90%.

2. Fix one tool list for all users. tools sits at position 0, so once it diverges there is no way to recover. Rather than splitting the list by permission, give everyone the full set and do the permission check in the execution handler. Reject unauthorized calls with an error result and the model will route around it on its own.

3. Make serialization deterministic. The cache key is the rendered bytes. If JSON key ordering wavers, logically identical content becomes a different prompt. Make the sort_keys option a habit, and sort things like search results by document ID before inserting them.

4. Treat conversation history as append-only. Editing or trimming the middle of the history invalidates everything after that point. Touch the third turn of a 200K-token conversation and 190K tokens are recomputed at list price. For things like mode switching, appending a system-role message after the history — supported on recent models — beats rewriting the system prompt.

5. With RAG (Retrieval-Augmented Generation), separate fixed documents from per-query retrieval results. Put retrieval results up front and everything after them is invalidated whenever the query changes. Retrieval results belong at the very end.

6. Subagents and forks should copy the parent’s prefix verbatim. In side calls like summarization or judging there is a temptation to build a “lighter” system prompt from scratch, but if it cannot ride the parent’s cache, that is a net increase.

That Said, Not Everything Breaks#

Invalidation is layered: a change invalidates its own layer and everything below it. Knowing this precisely keeps you from designing unnecessary workarounds.

There are only two things to worry about: tool definition changes and model switches.

TTL Is a Function of Traffic Interval, and Refreshes Are Free#

This is the most underrated fact in practice. The TTL restarts from the last hit, and that refresh costs nothing. A prefix that keeps receiving requests within five minutes is written once and stays alive essentially forever.

This is why designs that “reduce the number of distinct prefixes” compound. Cut prefixes from eight kinds to one and each prefix’s request interval drops to one eighth, which reliably lands you in the free-refresh zone.

With a Coding Agent, You Don’t Need to Think About It#

Everything so far applies to calling the API directly. So what about working through a coding agent like Claude Code or Cursor?

The short answer is that you don’t need to think about it. The harness, not you, is what assembles the request, so attaching cache boundaries is the harness’s job. And this is not optional. An agent resends the entire history every turn, so without caching the cost becomes unmanageable as the conversation grows. That is presumably why the Claude Code team titled their retrospective “Prompt caching is everything.”

The specifics below are for Claude Code. Details will differ by tool, but what struck me is that the structure Claude Code uses is exactly the strata model laid out above.

Claude Code also picks the TTL for you based on how you authenticate. On a Claude subscription it requests one hour automatically, so the cache survives a short break; on an API key or through a cloud provider, five minutes is the default.

Still, the Cache-Breaking Actions Are Worth Knowing#

Automatic does not mean independent of what you do. The actions below make the next turn slower and more expensive.

It is equally useful to note the actions that do not break the cache. Editing repository files, editing CLAUDE.md, changing output style, switching permission mode, invoking skills and commands, /recap, and spawning a subagent are all safe. Note though that for CLAUDE.md and output style, the cache survives but the change also does not take effect. They are read once at session start and held in memory, so the update lands after /clear or a restart.

A particularly useful contrast is /compact versus /rewind. If you have gone down the wrong path, /rewind beats /compact — rewind truncates back to a prefix that is already cached, whereas compaction builds a new one.

Another thing I hadn’t known: the cache is scoped per machine and per directory. The working directory, platform, shell, and OS version are baked into the system prompt, so sessions in different directories cannot share a cache. Worktrees of the same repository count as separate, too.

If You Use a Different Tool#

The command names above are specific to Claude Code. But the causes of cache invalidation are the same regardless of harness. Whatever coding agent you use, these three collapse the prefix.

• Switching models — caches are isolated per model, with no exceptions
• Changing the tool set — tool definitions sit at the very front of the prompt
• Rewriting history — summarizing, compacting, and mid-history edits all change the prefix

Find which actions in your tool correspond to these three, and you have that tool’s list of “expensive actions.”

And the Trap: Hit Rate Was Never the Goal#

Up to here the story was “raise the hit rate.” But while working out when to run compaction, I hit a strange result.

Compaction replaces history with a summary, so it destroys the cache. That makes compacting less often — and keeping the hit rate high — look like the cheaper choice. And by hit rate alone, it is.

The compacted side is 12 percentage points lower. But calculating the actual bill flips the result. (Claude Opus 5: $5 per million input tokens at list price, $0.50 per million for cache reads, $6.25 per million for cache writes.)

The uncompacted side has both a higher hit rate and more than double the caching savings — and a bill four times larger.

Why This Happens#

Because the 0.1× discount applies equally to both sides.

No matter how good the discount, the ratio between 300K and 30K stays 10 to 1. A multiplicative constant pulls both values down together; it cannot reverse their order. The 0.1× is a discount, not a ceiling.

The root of the problem is that hit rate is a ratio while cost is an absolute.

hit rate = cache read / (cache read + write + full-price input)

Grow the numerator and the hit rate rises — but the way to grow the numerator is “keep the history large.” In other words, the surest way to maximize hit rate is also the most expensive choice. An agent carrying 900K of history has a 99.5% hit rate and pays $0.45 per request.

So When Do You Look at Hit Rate?#

It resolves once you separate the conditions under which the metric is valid.

Of the guidelines above, freezing the system prompt, deterministic serialization, fixing the tool list, and copying the parent prefix into subagents all fall in the first category. They are about making identical content cacheable, so they are pure gain and hit rate works as a performance metric directly.

Compaction policy, RAG strategy, and session-length decisions fall in the second. They are decisions that adjust total context volume, so they must be judged by absolute cost per request, not by hit rate.

Hit rate is a “diagnostic metric for catching waste,” not a “target to maximize.” A low hit rate is a signal that the prefix is wobbling; a high one means nothing is wrong, not that things are optimal.

The Real Criterion for Compaction#

So how should you decide whether to compact? Not by hit rate, but by the cost of one compaction event against the accumulated savings that follow.

If the session will run eight or more turns past the compaction, compacting wins; if it ends before that, it is a net loss. Cumulatively the difference is starker. Without compaction, per-request cost rises linearly, so the cumulative cost over N turns scales with N squared; with compaction it resets in a sawtooth and scales roughly with N. The longer the session, the faster the gap widens.

Also, the Cache Does Not Save You Throughput#

There are costs that don’t show up in dollars. Cache-read tokens still count 100% against your per-minute throughput limit. The 0.1× applies only to the bill.

On a 2-million-tokens-per-minute limit, carrying 900K of history caps you at two requests per minute. The hit rate looks flawless at 99.5% while the agent sits waiting. Here compaction is not a cost optimization but a way to reclaim throughput.

Add response latency and context drift on top of that, and the strategy of deferring compaction indefinitely pays a price in three directions.

Wrapping Up#

Splitting it into before and after digging in, it comes out like this.

And compressed into practical guidance, two sentences remain.

First, caching optimization is not a question of “what to cache” but of “what to put first.” Stack the prompt in layers by change frequency and block the code that breaks the stratigraphy, and marker placement follows almost automatically.

Second, caching improves “how much you saved,” while total context volume determines “how much you pay.” The two multiply, and the final number is dominated by the latter.

I set out to dig into caching alone, and it led to a much larger question: how to design context and how to run sessions. Perhaps that was the natural order all along.

References

• Anthropic — Prompt caching
• Claude Code — How Claude Code uses prompt caching
• Anthropic — Lessons from building Claude Code: Prompt caching is everything