A single agent turn can involve a dozen model calls, and every one of them resends the system prompt and the tool definitions. With a large tool catalogue, that’s tens of kilobytes per call before the user’s question even appears. Provider-side prompt caching — reusing the computed prefix of a request that matches a recent one — is what makes that affordable.
The thing that took me a while to absorb is that the cache doesn’t care about your intent. It cares about bytes. The prefix of your request is effectively an API between you and the provider, and very ordinary code can break it without anyone noticing.
A bug that’s easy to write
Here’s one. Somewhere between your agent and the provider sits a layer that logs requests — for debugging, cost tracking, audit. Someone wants the logs to be deterministic and easy to diff, so they normalise the tool list:
body["tools"] = sorted(body["tools"], key=lambda t: t["function"]["name"])With a fixed set of tools, this is harmless. Every request has the same sorted list.
Now add on-demand tool loading. The agent starts with a core set and loads more as the task needs them. You did the cache-friendly thing and appended new tools to the end, so the prefix up to the old last tool stays identical. Except the logging layer re-sorts. A newly loaded tool whose name starts with “a” jumps to the front, shifts every tool after it, and the cached prefix now ends right after the system prompt. It happens on every call after every load.
The symptom is subtle. Nothing errors. Costs go up and latency gets a bit worse. If you also shrank the initial tool list to save tokens, your dashboard can show the optimisation making things worse — cache coverage dropping by half or more — and it’s tempting to conclude that dynamic tool loading just doesn’t pay.
The fix is to delete the sort, and leave a comment where it was explaining why. Ordering should be decided in one place — whatever assembles the tool list — and that place should guarantee a deterministic, append-only order. Every layer after it should treat the list as opaque.
Where dynamic context goes
The general rule: anything that changes between calls goes after anything that doesn’t.
- System prompt: frozen. No timestamps, no user names, no “current state”.
- Tool definitions: stable order, appended to and never reshuffled.
- Conversation history: grows at the end. Rewriting old turns breaks the prefix from that point.
- Dynamic context — the current time, a snapshot of the user’s data, loaded procedures, a status line: appended as trailing messages, last.
The classic cache breaker is a clock in the system prompt. “Today is Tuesday, 14:32” changes every minute, and with it the first few hundred bytes of every request. Put the time in a trailing message instead. Same information, and the prefix survives.
A cheap safeguard is a lint that runs before every call: scan the system prompt for markers that should never be there — “Today is”, a serialised workspace state, an id of the currently selected item — and record a warning if one shows up. It catches the regression where someone helpfully adds context to the wrong place.
One deliberate break
Compacting a long conversation — replacing older turns with a summary — changes the history prefix by design. That’s fine, as long as it happens rarely. Store the summary and reuse it across calls until the conversation grows past the threshold again. Regenerating it on every call would make it a moving prefix, and you’d lose the cache on exactly the long conversations where it helps most. The compaction post covers the rest.
Measure what you offered the provider
You can’t fix cache regressions you can’t see, and provider dashboards usually show a single aggregate. I’d add two kinds of per-call telemetry.
Component hashes. For every request, record the byte size and a hash of each part: system messages, tools, history, each individual message, and parameters. For tools, record two hashes: one of the list in its actual order and one sorted. If the sorted hash matches the previous call but the ordered one doesn’t, something reordered your tools. That pair would have caught the bug above on the first call. Hashing the raw request bytes at a few fixed lengths (say 256 bytes, 4 KB, 16 KB, 64 KB) also tells you roughly how far into the request two calls agree.
A per-run comparison. Compare each call with the previous one in the same run:
{
"system_changed": False,
"tools_changed": False, # order-sensitive
"matching_leading_messages": 14,
"history_rewritten": False,
"history_shortened": False,
}Be clear about what this is. It tells you whether you offered the provider a stable prefix. Whether the provider used it is a separate fact, read from the usage data it returns. Routing matters too: with some providers, sending a stable cache key or session hint makes it more likely that requests with the same prefix land where the cache already is. Treat that as a hint, never a guarantee, and judge it by observed usage.
Missing data isn’t zero
Providers report cached tokens under different names — cached_tokens inside prompt token details on OpenAI-style APIs, separate cache read and write fields on Anthropic’s — and sometimes not at all: a stream that ended early, a proxy that dropped the usage block. Normalise the field names into one record, and keep a separate flag for whether cache data was reported.
A call with no cache data is unreported. It isn’t a 0% hit and it isn’t a 100% hit. If you fold those calls in either way, your headline number moves for reasons that have nothing to do with caching.
Be careful with savings numbers
The tempting report is a single line: “caching saved X%”. You can compute something like it — price the same observed tokens as if none had been cached, and compare with what you actually paid. That’s a reasonable discount figure for one model and one price list. It isn’t evidence that a particular code change reduced spending, because the workload changed too.
Per-session numbers also vary enormously. A one-question session never warms the cache. A long research session with a stable prefix lives in it. You can easily see sessions near 0% and near 95% in the same week, and an average describes neither. I’d report the distribution, and the share of calls with unreported usage, before any single percentage.
When fewer tokens costs more
Finally, a trade-off worth testing rather than assuming. Loading tools on demand does shrink each request. But the agent then spends calls discovering and loading tools, and each extra call resends the whole prefix. In a fair comparison on frozen inputs, deferred loading can come out more expensive in total input than attaching the full tool set for the task, even with smaller requests.
My default now: attach the full, stable tool set for the kind of task at hand, keep on-demand loading as an escape hatch, and let the cache do its job. It’s less clever than the alternative. Whether it’s cheaper is something you can measure, as long as you measure the whole task and not bytes per request.
Both OpenAI’s and Anthropic’s caching docs are worth reading for the provider-specific rules on minimum prefix length and cache lifetime.