Context compression looks like an easy win. Agents send a lot of tokens, so send fewer and pay less.
In practice, many teams turn on a compressor and see the bill stay flat, or go up. The token count drops. The invoice does not. The reason is the provider’s prompt cache, and the fix is to treat the cached prefix as something you must not touch.
Agents resend everything
Model APIs are stateless. The provider keeps no memory of the last call, so an agent sends everything again on every call: the system prompt, the tool definitions, each earlier turn and every tool output it has collected.
A coding agent that reads files, runs tests and searches logs adds a large block of tool output on almost every turn. The request grows with each turn, and each turn pays again for everything before it. Late in a session, the part the model needs for the next step is small compared to the history around it.
That history is the obvious thing to compress. Old tool output, files read twice, passing tests and healthy log lines are all good targets. The problem is how most compressors remove them.
How the provider bills a request
Providers cache the start of each request: the cached prefix. When the next request begins with the same bytes, the provider reads that prefix from cache and bills it at a small fraction of the normal input price.
The first time the provider stores a prefix, it bills a cache write, which costs the normal price or more. Output costs the most of all. So a provider does not bill tokens. It bills each token by its class.
A well-behaved agent session is therefore mostly cheap cache reads, plus a short new tail on each turn. Late in a session, most of the request is already cached. Removing cached tokens saves little, because they were cheap to begin with.
Where naive compression goes wrong
A naive compressor summarizes or trims the history again on every call. It reports fewer tokens, but it loses in three ways.
- Broken cache. A cache match is exact and positional. If one byte changes early in the request, the cache stops matching at that byte, and everything after it is billed again as a cache write. A compressor that re-processes history on each turn makes slightly different bytes each time.
- Lost bytes. An edit tool fails on paraphrased code. The agent reads the file again, and the removed tokens come back.
- Extra calls. A summarizer is itself a model call, and you pay for it on every turn.
The result is a smaller request on paper and a larger bill in practice. The saving that counts is the change in the session bill for the same result, not the number of tokens removed.
Keep the prefix stable
The Anyray compression engine starts from the other end. It treats the cached prefix as something it must protect, and judges each edit by what it does to the bill over the whole session. Four rules follow from that.
1. Determinism
The same input bytes always give the same output bytes. No edit depends on time, randomness or state that drifts between turns. Content the engine compresses on one turn reaches the provider as the same bytes on every later turn, so the cache keeps matching.
2. Pin and replay
The gateway remembers, per session, each edit the engine made. Later turns replay the same edit. This includes turns that skip the engine, for example after a timeout. The cached prefix does not drift.
3. Payback test
Content that is already cached is edited only when the saving over the rest of the session is larger than the one-time rewrite plus a possible read-back. Short sessions are left alone, because the rewrite would never pay for itself.
4. Stable start
A dedicated edit keeps the start of the request stable so the provider can reuse it. Cache-sensitive edits stand aside when they would move an unstable byte into the cached prefix.
Where a provider offers its own way to clear old content before billing, the engine uses it. It sets the trigger so the provider clears only when the context window runs low, and only as much as it needs.
Removed is not lost
Cache safety is half the problem. The other half is the agent itself. An agent acts on exact bytes, and the provider validates the structure of tool calls and tool outputs.
So the engine keeps exact bytes where they matter. Code the agent is editing, errors, the current turn and the user’s own words pass through unchanged. When an edit removes an old tool output, it stores the original and leaves a marker with a handle. If the agent needs that content later, it fetches the exact bytes back with one tool call. Most lossy edits become deferred reads: the agent pays for the content again only on the turns that use it.
What to measure
If you compress agent traffic today, check three things before you trust the savings number:
- Cache read share. If it falls after you turn compression on, the compressor is breaking your prefix. cacheLint shows the first segment that changed.
- Cache writes per turn. They should cover only the new tail, not the history.
- Read-backs and retries. If the agent fetches files again or retries edits more, the compressor is removing bytes it needs.
Then compare the session bill for the same task, with and without compression. That is the only number that tells you what you saved.
Where to go next
- The compression engineThe four kinds of waste and the edits that remove them.
- How Anyray worksWhere the gateway sits and how it fails open.
- Optimizer docsHow just-in-time optimization runs.