Putting a gateway in front of every model call is a fair thing to worry about. If it goes down, do your agents go down with it?

They should not, and with Anyray they do not. Every layer we add to the request path has one rule: if it cannot do its job safely, it gets out of the way. A failure costs you the savings on that request. It never costs you a wrong answer or a failed call.

This post walks through what fails, what happens next, and the one detail that most fail-open designs miss: the prompt cache.

Three layers, three ways out

An Anyray request passes three things that can fail on their own: a single transform in the optimizer pipeline, the optimizer service as a whole, and the gateway. Each one fails open in its own way.

Figure 1. The three layers that can fail, from the inside out. Each one has a defined way out, and none of them ends in a failed request.

A transform that throws is skipped

The optimizer runs an ordered pipeline of small transforms. Each one handles one kind of waste and checks its own result. If a transform throws, the pipeline skips it and passes the request on to the next one. The other transforms still run.

The same rule applies when a transform is not sure. If it cannot parse the code or JSON it is about to edit, it keeps the original. If its own check fails, it discards the edit.

The optimizer is advice, not a gate

The gateway does not route traffic through the optimizer. It asks the optimizer how to make the request cheaper, and waits a fixed time for an answer. The optimizer sits beside the request path, not in front of it, and it never calls the provider itself.

The gateway waits up to 1200 ms. Requests with more than about 30k tokens of content get up to 2.5 times that, because the work grows with the content. On any timeout or error, the gateway forwards the original request to the model.

Figure 2. The hook budget for one request. Large requests get up to 2.5 times the base budget. The optimizer starts to finish early at the halfway mark, so most turns still get optimized.

The optimizer also watches the clock. At the halfway mark it starts to stand down the strategies it can drop safely, so it can finish the turn with what it has. A trace that says budget_exhausted means some strategies were skipped, not that the turn stopped optimizing.

When the optimizer cannot be reached at all, the gateway stops calling it after a few failed tries and forwards requests straight away. It tries again after a second, then backs off up to 30 seconds, and resumes on the first request that gets through. A slow optimizer is not treated as down. It keeps the normal budget.

Fail open, but keep the cache warm

Here is the detail that matters for your bill. Say the optimizer compressed an old tool output on turn 2. The provider cached the compressed bytes as part of the prefix. On turn 3 the optimizer times out.

If the gateway now sends the original, uncompressed request, the bytes at that position change. The cache stops matching there, and everything after it is billed again as a cache write. A plain fail-open design saves you from a failed call and then charges you for it.

Figure 3. Illustrative. If a failed turn sent the original bytes, the cached prefix would break and the rest of the request would be billed again. The gateway replays the earlier edit instead, so the cache stays warm.

So the gateway keeps a short memo of each session’s earlier rewrites. When a turn skips the optimizer, the gateway replays those edits, and the provider sees the same bytes it cached. The memo lives in one process, holds only bytes the provider already got, lasts one hour, and is gone on restart. It is never written to disk or exported.

When the gateway itself is down

If only the optimizer is down, leave your clients on the gateway. It starts without waiting for the optimizer and keeps routing, recording spend and attributing usage.

If the gateway is down, Anyray Connect moves Claude Code and Codex to each developer’s own subscription. At the default setting this takes about three minutes. Connect moves them back once the gateway has been healthy for ten minutes. A machine whose IT policy pins the gateway route does not switch, and you can turn fallback off for your organization.

One edge case: a reply that was already running when the gateway went down stays on it. Press Esc and send the message again, and the new request goes direct. You can also switch by hand with anyray-connect fallback.

The gateway also absorbs provider hiccups. It retries transient upstream errors and rate limits, honors the provider’s retry-after, and can fall back across providers when you configure it.

Budgets fail open too

Spend controls are the other place a gateway can block work. Anyray’s per-user caps and budgets are soft brakes. If the budget store is down, slow, or not warmed up yet, the request goes through. A budget outage must not stop your agents or your CI.

The trade-off is honest: requests already in flight can overshoot a ceiling by their combined cost. We think that is the right call for a brake that protects spend, not security.

What a failure costs you

Put together, the worst case on any single request is simple: you paid full price for it. You never get a worse answer, a dropped call, or a stalled agent because the layer meant to save you money had a bad day.

You can also turn any optimization off at runtime, with no redeploy, through the admin settings API. Changes are audit-logged and can be scoped by endpoint, model, user or team.

Where to go next

← Back to blog