Most AI gateways handle routing, keys and spend dashboards. That is useful, but it is not where the bill comes from.
The bill comes from tokens the model does not need. Agents resend old tool output, long logs, repeated instructions and bulky JSON on every call, and each call pays for all of it again. Anyray removes that waste before the request leaves your network, without breaking the provider’s prompt cache.
Where Anyray sits
Anyray is a gateway that runs in your own cloud. Your tools send requests to it instead of straight to the provider. The gateway optimizes each request and forwards it to the model APIs you already use.
Three things follow from that position:
- Your content stays with you. Requests are optimized inside your network. No new vendor sees prompt content, and provider keys stay on the gateway.
- Your tools do not change. You change one base URL. For local coding tools, Anyray Connect sets it for you.
- Spend has an owner. Each request is tied to a user, team, project and model, so cost becomes a report instead of a guess.
Start in shadow
Optimization has three modes, and you set them per route. A sensitive route can stay off while the rest run live.
- Off. Requests pass through unchanged.
- Shadow. The optimizer runs and reports what it would save. The model still gets the original request.
- Live. The model gets the optimized request.
Shadow shows the saving on your own traffic before one byte the model reads changes. Turn on live when the numbers clear your bar.
What the engine cuts
The engine removes waste, not meaning. It keeps the parts that matter and drops or shortens the rest. It does not ask a model to rewrite your context.
Some content always passes through unchanged: tool call arguments, the system prompt, code, errors, the current turn and the user’s own words. Tool output keeps its JSON structure, so the agent can still parse it.
The rest is fair game: repeated blocks, extra whitespace, stale tool output and long logs where only a few lines matter. When an edit removes content, Anyray keeps the original so the agent can fetch it back. How retrieval handles make lossy compression safe explains how.
Every edit is cache-safe
Providers cache the start of each request and bill it at a fraction of the normal price. A compressor that makes different bytes on each turn breaks that cache, so the bill goes up while the token count goes down. Why context compression breaks prompt caching covers this in detail.
The engine follows three rules to avoid it.
1. Same input, same output
How a message is compressed depends only on its own bytes. The same message gives the same result on every turn, so content you already sent stays a cache hit.
2. Cut in batches
The engine does not rewrite the history on every turn. It moves forward in steps, so the provider rebuilds its cache rarely, not on every message.
3. Cut only when it pays
Each edit is priced against the provider’s cache rules. If rewriting cached content costs more than the tokens it saves, the engine skips the edit.
It fails open
Compression must never take production down. If the optimizer errors or the gateway is unavailable, the request passes through unchanged and the model still answers.
The gateway runs as a container in your account and talks only to the model APIs you already use. When something goes wrong, the right behavior is boring: traffic flows, and you look into it later.
Where to go next
- The compression engineThe kinds of waste and the edits that remove them.
- Gateway docsWhere Anyray sits on the request path.
- Optimizer docsHow the modes and per-route settings work.
- RTK alternativeAnyray vs RTK on the same benchmark tasks.
- Headroom alternativeAnyray vs Headroom on the same benchmark tasks.