I Dug Into Prompt Caching and Found That Hit Rate Isn't the Goal

August 5, 2026

LLM, Prompt Caching, Claude, Coding Agent, Cost Optimization, Context Engineering

I Dug Into Prompt Caching and Found That Hit Rate Isn’t the Goal#

2026-08-05

Prompt content stacked like geological strata by change frequency, with the lower layers reused from cache

It Started With “Where Does This Actually Happen?”#

I knew prompt caching reduced costs. My understanding went about as far as: when you send the same system prompt repeatedly, it gets cheaper from the second request onward.

But the more I thought about it, the stranger it seemed. If caching happens in some relay layer outside the model, then from the model’s point of view nothing is saved. And yet the savings are said to reach 90%. Would that mean most of the cost of using an LLM (Large Language Model) comes not from the model itself but from the layer around it?