I Dug Into Prompt Caching and Found That Hit Rate Isn't the Goal

Wed, 05 Aug 2026 00:00:00 +0900

I Dug Into Prompt Caching and Found That Hit Rate Isn’t the Goal#

2026-08-05

It Started With “Where Does This Actually Happen?”#

I knew prompt caching reduced costs. My understanding went about as far as: when you send the same system prompt repeatedly, it gets cheaper from the second request onward.

But the more I thought about it, the stranger it seemed. If caching happens in some relay layer outside the model, then from the model’s point of view nothing is saved. And yet the savings are said to reach 90%. Would that mean most of the cost of using an LLM (Large Language Model) comes not from the model itself but from the layer around it?

Prompt Caching on Ted Factory

I Dug Into Prompt Caching and Found That Hit Rate Isn't the Goal

I Dug Into Prompt Caching and Found That Hit Rate Isn’t the Goal#

It Started With “Where Does This Actually Happen?”#