A cache that never reads is not a cache. It is a surcharge.
I run a small personal chat application against a set of hosted model deployments. It supports prompt caching, because a long system prompt and a growing conversation are exactly the shape caching exists for. For weeks I assumed the feature was working, because nothing about it ever failed.
Then I put the cache token counts in the test report, and one deployment turned out to have been quietly charging me a premium on every single turn.
The asymmetry that makes caching worth doing
Prompt caching does not make the cacheable part of a request free. It makes it cheaper to read than to write.
On the deployments I use, against the base input rate:
- writing a cache entry bills at 1.25×;
- reading one back bills at 0.1×;
- everything not covered by the cache — the new user message, the response — bills as normal.
That asymmetry is the whole mechanism. You accept a 25% premium once, to buy a 90% discount on every subsequent turn that reuses the same prefix.
One read pays for the write
The arithmetic is worth doing explicitly, because it sets how much attention the read side deserves.
For a prefix of a given size held constant across N turns:
No caching: N × 1.00
With caching: 1.25 + (N − 1) × 0.10Those cross at N ≈ 1.28. Which means a single read pays for the write, and from the second turn onwards caching is winning. It is one of the few optimisations where the break-even arrives almost immediately.
The same arithmetic run over four turns shows why the failure case is so unpleasant:
| Four turns | Cost of the prefix | Against no caching |
|---|---|---|
| No caching | 4.00 | — |
| 1 write, 3 reads | 1.55 | 61% cheaper |
| 4 writes, 0 reads | 5.00 | 25% more expensive |
The last row is not a degraded optimisation. It is the worst of the three options, and it is what a cache that never reads back produces.
Three deployments, one code path, two behaviours
My application sends every model through the same adapter. Same request construction, same cache breakpoints, same effort level, same thinking configuration. The only difference between deployments is the model name.
Across two live runs, the per-turn cache counters looked like this:
| Deployment | Cache writes | Cache reads |
|---|---|---|
| A | 1 | 3 |
| B | 1 | 3 |
| C | 4 (≈5,640 tokens each) | 0 |
A and B are textbook: write the prefix once, read it back on every subsequent turn. C wrote a fresh entry on all four turns and never once read one back.
So C was paying 1.25× on every turn and collecting the 0.1× never. Strictly worse than sending no cache instruction at all.
Accepted is not the same as effective
The reason this survived for weeks is the reason it is worth writing about.
Nothing errored. The cache writes were accepted, which rules out the most obvious explanation — that the cache control instruction was being rejected or ignored outright. The API returned 200. The responses were correct. The application behaved exactly as intended. A monitoring check on status codes, latency or output quality would have reported a perfectly healthy system, because by every one of those measures it was one.
The only signal was in two numbers that nothing was looking at.
A provider feature that returns success has not necessarily done anything. If the benefit is metered, the meter is the test.
What made it visible
Two changes, neither of them large.
The contract test reports cache tokens per turn instead of asserting a status. A test that checks a multi-turn conversation succeeds will pass whether the cache is working perfectly or not at all. The useful assertion is on the counters:
def test_multi_turn_reads_the_cache(client):
"""A conversation that reuses a prefix must read it back, not rewrite it."""
usage = [client.send(turn).usage for turn in FOUR_TURN_SCRIPT]
writes = sum(u.cache_creation_input_tokens for u in usage)
reads = sum(u.cache_read_input_tokens for u in usage)
# One write, then reads. More than one write means the prefix is moving.
assert writes > 0, "cache control had no effect at all"
assert reads > 0, f"{writes} writes and no reads: paying 1.25x for nothing"A pricing helper computes what caching actually saved, and is allowed to go negative. Costing a run as “input tokens × rate” hides the whole problem, because it never separates the two buckets:
def cache_saving(usage, input_rate):
"""Positive when caching paid. Negative when it was a surcharge."""
uncached = (usage.cache_creation_input_tokens
+ usage.cache_read_input_tokens) * input_rate
actual = (usage.cache_creation_input_tokens * input_rate * 1.25
+ usage.cache_read_input_tokens * input_rate * 0.10)
return uncached - actualA negative return value is the entire finding. It is not an edge case to guard against — it is the number that tells you the feature is costing you money.
Three things that silently move a prefix
The failure above was not self-inflicted, but three adjacent ones were, and they are much more common. All three change the cached prefix on every turn, so every turn writes a new entry and reads nothing. None of them error.
A clock in the system prompt. Injecting the current time so the model knows the date rewrites the prefix on every single request. The fix is to make the time a tool the model calls when it needs it, which also stops the model asserting a timestamp it was handed but never asked for.
A setting change part-way through a conversation. Effort level, thinking configuration, temperature and the system text itself all sit ahead of the messages in the cache key. Change one on turn three and the first two turns stop being reusable — invisibly, because the conversation carries on working.
Tool definitions serialised in a non-deterministic order. If the tool schema is assembled from a dictionary or a set, the order can differ between processes. The definitions are identical in meaning and different as bytes, so the prefix misses. Sort them.
What I changed
The flag is now per deployment, not global. One entry behaving differently is not a reason to give up the 61% saving on the others, and a global switch forces exactly that choice.
Alongside it, a governance note in the model registry records the measurement, the date, both runs, the comparison against the two deployments that behaved correctly, and — the part I would previously have skipped — the condition under which to turn it back on. A disabled feature with no reversal condition becomes permanent by accident.
A test asserts both that the flag is off and that the note explaining why is present, so a future tidy-up cannot silently re-enable it without meeting the evidence bar that disabled it.
What this is not
This is one resource, one date, two runs.
The same code path read back normally on two other deployments, which makes a fault in my request construction unlikely but does not exclude something specific to that deployment, region or resource. I have not reproduced it elsewhere, and I have not tried to establish whether it would still happen today.
So this is a report of a measurement and a method for catching it — not a general claim about a model. If you are choosing between deployments on cost, measure your own; the point of the article is that you can, cheaply, and that nothing will tell you if you do not.
Lessons
Price both buckets separately. A single input-token figure cannot distinguish a cache that is paying for itself from one that is a 25% surcharge.
Assert the metric, not the status code. Every measure that a normal health check looks at was green throughout.
Make the flag per deployment. Model behaviour is not uniform, and a global switch turns one bad entry into a site-wide regression.
Record the condition for reversal. Otherwise a temporary mitigation quietly becomes architecture.
Watch for anything that moves the prefix. A clock, a mid-conversation setting change and an unsorted tool list all produce the same symptom as a broken cache, and all three are your own doing.
This describes a personal project of mine, not any employer system. The figures are from my own tool and its public repository. Endpoint names, resource names and keys are not reproduced here.
