Every agent request is a small decision problem. The session is long, the prompt is full of things the model has already seen, and some of those things can be shortened or removed. Some should not be touched at all.
The usual workarounds are fixed rules: always trim tool output, always summarize old turns. A fixed rule is wrong somewhere. It works on one team’s sessions and costs another team money. Anyray does not ship fixed rules. It ships a decision engine that makes a fresh call on every request, and a learning loop that checks those calls against each org’s own traffic.
This post is a tour of both. It covers what an operator can see and rely on, and leaves the recipe out.
One decision, every request
The engine sits beside the request path. For each request, the gateway hands it the facts of the lane: the model, the provider, where the prompt cache ends, and what the calling client can do. The engine then works through a short sequence.
Three properties come out of this shape, and they matter more than any single edit:
- Nothing is forced. An edit runs only if the request looks right for it and the lane can take it safely. A client that cannot read a handle never gets an edit that needs one.
- Doing nothing is a valid answer. When no edit qualifies, the request goes out as it came in.
- It fails open. If the engine is slow or down, the gateway forwards the original request. The engine can only make a request cheaper, never block it.
Same bytes next turn
A provider’s prompt cache pays off only when the start of the prompt is byte for byte the same as last turn. An edit that is decided fresh on every turn, even a good one, can change those bytes and cost more than it saves.
So the engine remembers. Once an edit is settled for a session, later turns replay it exactly instead of deciding again. The same part of a conversation is never judged one way on turn ten and another way on turn eleven. The cache post covers why this matters. For this post, the point is simple: a decision is not just made, it is kept.
Every edit has to earn its place
Here is the part that makes it learning rather than rules. An edit that changes the bytes the model sees is on probation on every new deployment, and on every client tool it sees.
While an edit is unproven, it runs only on a small share of sessions. The rest run without it, and a session never changes groups. The engine compares the two groups, whole session against whole session, on what each cost to finish the work. Then it acts on the result:
- It wins clearly. The edit is promoted and runs on all eligible traffic.
- It loses clearly. The edit is demoted. Repeated losses switch it off.
- It is unclear. Nothing changes. No verdict is a verdict.
The standard is pass-through. An edit that cannot beat forwarding the request unchanged does not stay on. We would rather ship a smaller saving we have proven than a bigger one we have only argued for.
Cost is not the only test. A cheaper session is not a win if the work got worse, so a quality check runs beside the cost check. It compares the two groups on four signals at 95% confidence: turns per session, output tokens per request, error rate and latency. A regression needs a result that is both significant and larger than 10%. A real but trivial move does not count against an edit.
Learned per tool, per team
An org’s traffic is not one thing. Claude Code, Codex and a script in CI behave differently, and so do people in the same team. An edit that helps one can hurt another.
So the policy has three levels. It starts at the org, narrows for each client tool, and narrows again for a single user or agent key. A narrower level changes only the settings its own evidence supports, and takes everything else from the level above it. A tool with enough sessions is judged on its own, so a loss in one client tool turns the edit off for that tool only. The defaults an org starts with are not the settings it ends with.
| Edit | Coding agent | SDK job | Chat app |
|---|---|---|---|
| Compress old tool output | On | On | Off |
| Trim long logs | On | Probation | Off |
| Prune tool list | Probation | On | Off |
The table is illustrative, but the shape is real: the same edit can sit in different states for different tools, and the console shows the state of each one.
Learning from the model’s own reaction
Cost is one signal. The other is whether the model had to take something back.
When an agent fetches back a handle, re-runs a tool whose output was shortened, or retries after an edit, that is the model saying the cut was too deep. The engine counts these per edit and per tool, and weighs them against what the edit saved. If an edit keeps getting taken back, it is suppressed for that tool. After a cool-off, it is tried again on a few sessions. If the rate has dropped, it comes back.
Inside a single session the reaction is faster still: once the agent fetches something an edit removed, the engine cuts less for the rest of that session. The handles post shows what that looks like in practice.
What the engine learns from
The loop does not read the work. It reads how the work goes. Five kinds of signal feed it:
- Usage patterns. How long sessions run, how fast context grows, and how often an agent goes back to old content.
- Sessions. What a whole session cost to finish, with an edit and without it.
- Tools. Which client tool sent the request, and what that tool can do with a shortened result.
- Models. The price of each model, and where its prompt cache starts and ends. The same edit can pay on one model and lose on another.
- User behaviour. How one person or agent works: short tasks or long ones, and how often they return to earlier output.
All of it is metadata. The record behind each decision holds counts, sizes, costs, the edit that ran and the verdict. It holds no prompt text and no response text, and nothing is learned from content.
Why it matters
A compression engine is only as good as its worst decision. One right call is not enough: the calls must stay right as agents, models and workloads change. That is why the engine measures itself against doing nothing, on real sessions, all the time.
The result is visible. The session gate table in the console lists each edit, each tool, and the state each has earned.
Where to go next
- Optimizer docsThe pipeline, its settings and its decision record.
- Guardrails docsGate states, the regression guard and how to read a verdict.
- The compression engineThe edits the decision engine chooses between.
- Fail-open by designWhat happens when any layer has a bad day.