A broken prompt cache does not throw an error. The model answers the same way, the agent finishes the task, and every log line looks normal. The only sign is on the invoice: requests that should read the history from cache write it again, at full price or more.

We run Anyray in front of agent traffic, so we see this from both sides. Sometimes the client breaks its own cache. Sometimes an edit could break it. We needed a way to see every break, find the exact byte where it started, and say who caused it. That sensor is cacheLint, and this post explains how it works.

Why prompt caching fails silently

Providers cache the start of each request. When the next request begins with the same bytes, the provider reads that prefix from cache at a small fraction of the input price. The match is exact and positional. If one byte changes, the cache stops matching at that byte, and everything after it is billed again as a cache write.

Agents send the whole conversation on every turn, so the prefix is long and almost all of it should be a cache read. Many small things break it:

  • Tools in a new order. The client builds the tool list from a map or a plugin registry, and the order changes from turn to turn.
  • A live value in the system prompt. A timestamp, a request id or a git status at the top of the prompt changes on every call.
  • An edited early message. The client rewrites, trims or reformats history it has already sent.
  • An edit in flight. A transform in the gateway changes bytes that the provider already cached.

Each one is easy to fix once you know it is there. The hard part is knowing. A cache write rate tells you that something broke. It does not tell you what, where, or whose change it was.

What cacheLint compares

cacheLint is a read-only strategy in the Anyray optimizer. Its id in the strategy config is cache_lint. It runs before any transform, so it measures the request exactly as the client sent it. It changes nothing. On each turn, it compares the request with the previous turn of the same session and the same model.

It reads the request in the order the provider caches it: tool definitions first, then the system prompt, then the messages. It watches three regions:

  • Tools. The full list of tool definitions.
  • System. The system prompt, or the leading system message on OpenAI-style requests.
  • Message head. The earliest messages of the conversation. The live turn at the end is left out, because it is new on every call by design.
Figure 1. The segments cacheLint reads, in provider cache order. Illustrative. Each segment is reduced to a hash, and the live turn is left out.

Before it hashes anything, it removes the cache_control markers. A marker moves the point where the provider stores a cache entry, but it does not change the prompt bytes. Without this step, a client that moves its breakpoint forward on each turn would report a break on every turn.

Finding the first changed byte

Knowing that the system prompt changed is useful. Knowing where the prefix stopped matching is more useful, because that is the number the provider bills on.

cacheLint splits the request into segments: one per tool definition, one per system block, and one per message. It hashes each segment with sorted keys, so two JSON objects with the same content give the same hash. Then it carries a rolling prefix hash along them. Each value depends on the segment and on every segment before it.

To compare two turns, it walks both rolling hashes from the start. The first position where they differ is the divergence point. Everything before it is what the provider cache can reuse. Everything after it is billed again. The kind of the first changed segment names what broke the cache.

Figure 2. Two turns of one session. Illustrative. The rolling hashes agree for five segments, then split at the second system block. Each segment after the split is a cache write, even though most of them did not change.

The result is one line per break, in the decision log for that request. It says which regions changed, how many segments the turn kept from the last one, and the kind and position of the first changed segment. A divergence after the measured head counts too. An edited message later in the history breaks the cache as surely as an edited first message.

Why the tools changed

Tool changes are a common cause, and the right fix depends on the type of change. So cacheLint hashes the tool list three ways: in sequence, as an unordered set, and as an unordered set with descriptions removed. Comparing the three gives one of three answers.

Figure 3. The three answers cacheLint gives for a tool change. Illustrative. Each one points to a different fix in the client.

An order change is the cheapest to fix and the easiest to miss, because the tools look the same in any log. A description change usually means a live value, such as a date or a count, sits inside the tool text. A set change is often a client that loads tools lazily, for example MCP servers that connect after the first turn.

Who broke the cache

A break report is only useful if it says whose change caused it. If an Anyray transform changed the prefix, that is our problem to fix. If the client changed it, the fix is in the client.

cacheLint remembers which strategies made an edit on the previous turn of the session, and it sees which ones made an edit on this turn. Each break gets one of two labels:

  • Explained. An optimizer strategy changed the request on this turn or the turn before. The break may come from our own edit, so we check it against our cache-safety rules.
  • Client volatile. No strategy changed anything. The client sent different bytes on its own.

Read-only strategies, including cacheLint itself, do not count as edits. A sensor that measures the request cannot be the cause of a break.

Removing false alarms

A sensor that cries wolf gets ignored. Most of the work on cacheLint after the first version went into cases that looked like breaks and were not.

One prefix per model

A provider keys its cache by model. Coding agents often send one session to two models, a large one for the main loop and a small one for side calls. Each model holds its own prefix, and the two never share a cache entry. An early version compared each turn with the previous turn of the session, whatever its model. Every model switch looked like a full break. Now each session and model pair has its own baseline.

Markers are not bytes

Moving a cache_control marker does not change the prompt, so the markers are stripped before hashing in every region, at any depth.

Headers the provider does not cache

Claude Code puts a billing header in the first system block, and its fields change on every request. On most routes that block is part of the cached prefix, so a change there is a real break and cacheLint reports it. On one route, Bedrock, we probed the provider and proved that the cache ignores those fields. There, the block is replaced with a constant before hashing. We add a route to that list only after a probe of its own. The counters also report a change to the first system block apart from a change to a later block, because a later block is a real prompt change on any route.

The real session

The sensor keys on the conversation id the gateway has verified, not on a field the client may reuse across conversations. Each tenant also has its own session limit. One busy tenant cannot push another tenant’s sessions out of memory and reset its baselines.

What it stores

cacheLint never stores prompt content. Per session and model, it keeps the region hashes, the rolling prefix hashes and the names of the strategies that last fired. It holds them in process memory only, with a size limit. The counters it exports hold only counts, positions and fixed labels. If something fails, the log line names the error class and nothing from the request. A tenant purge clears the stored hashes with the rest of the tenant’s state.

How to find what broke your prompt cache

Each break shows on the request trace in the console as a Cache lint decision with the verdict line. The optimizer config read returns a rolling summary for the tenant: turns compared, turns that diverged, divergences by kind, first or later system block, and tool changes by type. These counters belong to one optimizer replica, so use them as a signal, not as a fleet total.

Most results map to one fix:

  • Tool order. Sort the tool list before you send it.
  • Tool description or set. Keep live values out of tool text, and register every tool at the start of the session.
  • Later system block. Move timestamps, ids and status lines out of the system prompt and into the latest message.
  • Message. Find the client step that rewrites history it has already sent.
  • Explained. Tell us. An Anyray edit that breaks a warm cache is a bug.

A clean session shows no breaks after the first turn. Every break you remove turns history you were paying to write again back into cheap cache reads.

Is cacheLint on GitHub?

No. cacheLint is not open source. It runs inside the Anyray gateway, as part of the optimizer, and its code is in our private repository. Our public benchmarks repository reads its decisions as read-only measurements, but it does not contain the sensor.

To see cacheLint on your own traffic, read how the compression engine keeps the cached prefix stable, or get a demo.

FAQ

What is cacheLint?

cacheLint is a read-only sensor in the Anyray gateway. On each turn of an LLM agent session, it compares the tools, system prompt and early messages with the previous turn. It finds the first segment that changed, which is where the prompt cache broke, and says if the client or an Anyray edit changed it.

Why is my prompt cache hit rate low?

Usually the client changes bytes early in the request. Common causes are tools that arrive in a new order, a timestamp or id in the system prompt, and history that the client rewrites. One changed byte makes every token after it a cache write. cacheLint names the first changed segment, so you know which one to fix.

Does cacheLint change my requests?

No. cacheLint is a read-only sensor. It runs before any transform, changes nothing in the request, and stores only hashes and counts, never prompt content.

Where to go next

← Back to blog