Putting a gateway in front of every model call is a fair thing to worry about. If it goes down, do your agents go down with it?
They should not, and with Anyray they do not. Every layer we add to the request path has one rule: if it cannot do its job safely, it gets out of the way. A failure costs you the savings on that request. It never costs you a wrong answer or a failed call.
This post walks through what fails, what happens next, and the one detail that most fail-open designs miss: the prompt cache.
Three layers, three ways out
An Anyray request passes three things that can fail on their own: a single transform in the optimizer pipeline, the optimizer service as a whole, and the gateway. Each one fails open in its own way.
A transform that throws is skipped
The optimizer runs an ordered pipeline of small transforms. Each one handles one kind of waste and checks its own result. If a transform throws, the pipeline skips it and passes the request on to the next one. The other transforms still run.
The same rule applies when a transform is not sure. If it cannot parse the code or JSON it is about to edit, it keeps the original. If its own check fails, it discards the edit.
The optimizer is advice, not a gate
The gateway does not route traffic through the optimizer. It asks the optimizer how to make the request cheaper, and waits a fixed time for an answer. The optimizer sits beside the request path, not in front of it, and it never calls the provider itself.
The gateway waits up to 1200 ms. Requests with more than about 30k tokens of content get up to 2.5 times that, because the work grows with the content. On any timeout or error, the gateway forwards the original request to the model.
The optimizer also watches the clock. At the halfway mark it starts to stand down the strategies it can drop safely, so it can finish the turn with what it has. A trace that says budget_exhausted means some strategies were skipped, not that the turn stopped optimizing.
When the optimizer cannot be reached at all, the gateway stops calling it after a few failed tries and forwards requests straight away. It tries again after a second, then backs off up to 30 seconds, and resumes on the first request that gets through. A slow optimizer is not treated as down. It keeps the normal budget.
Fail open, but keep the cache warm
Here is the detail that matters for your bill. Say the optimizer compressed an old tool output on turn 2. The provider cached the compressed bytes as part of the prefix. On turn 3 the optimizer times out.
If the gateway now sends the original, uncompressed request, the bytes at that position change. The cache stops matching there, and everything after it is billed again as a cache write. A plain fail-open design saves you from a failed call and then charges you for it.
So the gateway keeps a short memo of each session’s earlier rewrites. When a turn skips the optimizer, the gateway replays those edits, and the provider sees the same bytes it cached. The memo lives in one process, holds only bytes the provider already got, lasts one hour, and is gone on restart. It is never written to disk or exported.
When the gateway itself is down
If only the optimizer is down, leave your clients on the gateway. It starts without waiting for the optimizer and keeps routing, recording spend and attributing usage.
If the gateway is down, Anyray Connect moves Claude Code and Codex to each developer’s own subscription. At the default setting this takes about three minutes. Connect moves them back once the gateway has been healthy for ten minutes. A machine whose IT policy pins the gateway route does not switch, and you can turn fallback off for your organization.
One edge case: a reply that was already running when the gateway went down stays on it. Press Esc and send the message again, and the new request goes direct. You can also switch by hand with anyray-connect fallback.
The gateway also absorbs provider hiccups. It retries transient upstream errors and rate limits, honors the provider’s retry-after, and can fall back across providers when you configure it.
Budgets fail open too
Spend controls are the other place a gateway can block work. Anyray’s per-user caps and budgets are soft brakes. If the budget store is down, slow, or not warmed up yet, the request goes through. A budget outage must not stop your agents or your CI.
The trade-off is honest: requests already in flight can overshoot a ceiling by their combined cost. We think that is the right call for a brake that protects spend, not security.
What a failure costs you
Put together, the worst case on any single request is simple: you paid full price for it. You never get a worse answer, a dropped call, or a stalled agent because the layer meant to save you money had a bad day.
You can also turn any optimization off at runtime, with no redeploy, through the admin settings API. Changes are audit-logged and can be scoped by endpoint, model, user or team.
Where to go next
- Guardrails docsQuality, reversibility and reliability, in detail.
- Why context compression breaks prompt cachingWhy the cached prefix must not change.
- How Anyray worksWhere the gateway sits in your stack.