Most AI gateways handle routing, keys and spend dashboards. That is useful, but it is not where the bill comes from.

The bill comes from tokens the model does not need. Agents resend old tool output, long logs, repeated instructions and bulky JSON on every call, and each call pays for all of it again. Anyray removes that waste before the request leaves your network, without breaking the provider’s prompt cache.

Where Anyray sits

Anyray is a gateway that runs in your own cloud. Your tools send requests to it instead of straight to the provider. The gateway optimizes each request and forwards it to the model APIs you already use.

Figure 1. The request path. Your tools point at one URL. Anyray optimizes the request inside your network and forwards it to the provider. If anything fails, the request passes through as it was.

Three things follow from that position:

  • Your content stays with you. Requests are optimized inside your network. No new vendor sees prompt content, and provider keys stay on the gateway.
  • Your tools do not change. You change one base URL. For local coding tools, Anyray Connect sets it for you.
  • Spend has an owner. Each request is tied to a user, team, project and model, so cost becomes a report instead of a guess.

Start in shadow

Optimization has three modes, and you set them per route. A sensitive route can stay off while the rest run live.

  • Off. Requests pass through unchanged.
  • Shadow. The optimizer runs and reports what it would save. The model still gets the original request.
  • Live. The model gets the optimized request.

Shadow shows the saving on your own traffic before one byte the model reads changes. Turn on live when the numbers clear your bar.

What the engine cuts

The engine removes waste, not meaning. It keeps the parts that matter and drops or shortens the rest. It does not ask a model to rewrite your context.

Some content always passes through unchanged: tool call arguments, the system prompt, code, errors, the current turn and the user’s own words. Tool output keeps its JSON structure, so the agent can still parse it.

The rest is fair game: repeated blocks, extra whitespace, stale tool output and long logs where only a few lines matter. When an edit removes content, Anyray keeps the original so the agent can fetch it back. How retrieval handles make lossy compression safe explains how.

Every edit is cache-safe

Providers cache the start of each request and bill it at a fraction of the normal price. A compressor that makes different bytes on each turn breaks that cache, so the bill goes up while the token count goes down. Why context compression breaks prompt caching covers this in detail.

The engine follows three rules to avoid it.

1. Same input, same output

How a message is compressed depends only on its own bytes. The same message gives the same result on every turn, so content you already sent stays a cache hit.

2. Cut in batches

The engine does not rewrite the history on every turn. It moves forward in steps, so the provider rebuilds its cache rarely, not on every message.

3. Cut only when it pays

Each edit is priced against the provider’s cache rules. If rewriting cached content costs more than the tokens it saves, the engine skips the edit.

Figure 2. One request through the engine. Illustrative. Exact bytes are protected first, and an edit ships only when it lowers the bill for the session.

It fails open

Compression must never take production down. If the optimizer errors or the gateway is unavailable, the request passes through unchanged and the model still answers.

The gateway runs as a container in your account and talks only to the model APIs you already use. When something goes wrong, the right behavior is boring: traffic flows, and you look into it later.

Where to go next

← Back to blog