Most “AI gateway” writeups stop at routing, auth, and spend dashboards.
That’s necessary. It’s also not where the bill is.
The bill is in unused tokens: repetitive tool dumps, bloated RAG chunks, OCR’d screenshots billed as vision, the same system instructions and JSON noise shipped on every turn. Marketing math says 40–70% of a typical request is waste. You still pay 100% of it, at full rate, every call, across every teammate and agent.
Anyray is self-hosted pre-model middleware. It sits between your users/agents and the LLM APIs you already use. On the hot path it can compress and optimize eligible context just in time, attribute spend to real work, and fail open so a bad day never becomes an outage.
This post is the architecture deep-dive: what Anyray is, where it sits, how optimization actually runs, and why we care about determinism and prompt cache as much as compression ratio.
The shape of the problem
A typical coding-agent turn isn’t a short prompt. It’s:
- a long system preamble
- conversation history
- tool results (logs, diffs, issue threads, search hits)
- retrieved chunks
- maybe an image that should have been text
Providers bill input + output. Agents make the input side explode. And because clients usually resend the full history each turn, yesterday’s waste becomes today’s prefix — and tomorrow’s cache bill.
Cutting tokens after they hit the model doesn’t help. The waste has to leave before the request leaves your network for the provider.
That’s the “pre-model” claim. Not another chat UI. Not a new model. Middleware in the path you already have.
Where Anyray sits
The gateway is that box: routing, auth, and the request path your tools already speak.
Design constraints we won’t compromise on:
- Self-hosted — runs inside your cloud. Requests are optimized in-network, then forwarded to endpoints you already trust. No new vendor in the data path for content.
- Seamless — apps point at the gateway (via something like Anyray Connect for local tooling). Workflow doesn’t become a science project.
- Fail-open — if optimization errors or Anyray is unavailable, traffic can go straight through untouched. Compression is never allowed to take production down.
- Attribution — savings and spend broken down by project, team, model, and intent (debugging vs feature work vs review), so finance gets an ROI conversation, not an invoice argument.
Tagline we actually mean: stop paying for unused tokens.
Optimization is opt-in, not magic
On the gateway, prompt optimization is a controlled mode — typically off, shadow, or live. The optimizer docs cover how that hook sits on the request.
| Mode | What happens |
|---|---|
| Off | Pipeline skipped. Default-safe. |
| Shadow | Full pipeline runs; you see would-be savings; body is not mutated. |
| Live | Compressed body is written back onto the request before the provider call. |
Shadow exists so you can measure ROI on real traffic before you change a single byte that reaches the model. Live is what you turn on when the numbers clear the bar for a route class.
Per-route overrides matter: one flaky route can stay off while others run live, without rewriting global defaults.
The compression engine: Frozen-Frontier Extractive Compression (FFEC)
Under the hood, Anyray’s optimizer is built around FFEC — Frozen-Frontier Extractive Compression.
The important properties are not “we use an LLM to summarize harder.” They’re systems properties:
1. Extractive, not abstractive rewrite
We keep important words and spans; we delete or normalize the rest. Tool-use arguments, images, opaque blocks, and system-role text are protected. Tool results can be compressed at the value level while preserving JSON structure. Code fences, inline code, URLs, and tables are segmented as protected structure, not prose to mangle.
That’s how you avoid “the compressor rewrote my stack trace into poetry.”
2. Pure function of message bytes → cache-stable by construction
Clients resend history. Providers cache prefixes. If compression is non-deterministic, you silently burn cache writes and destroy the savings you thought you earned.
So compression of a message is designed as a pure function of that message’s bytes (plus a versioned policy / scorer artifact). Same input → same output bytes across turns. Recompressing an older message yields identical bytes; history that already compressed stays stable.
3. A monotone “frontier” that batches cache invalidation
You can’t aggressively rewrite the entire prefix every turn without thrashing cache. FFEC uses a monotone, batched frontier: as the conversation grows, the compressible window advances in controlled steps (batch size K), so cache invalidation is bounded — ideally one meaningful rewrite cadence per batch, not chaos every message.
Frozen vs mutable protection is informational for operators; purity means recomputation stays identical either way.
4. Importance scoring with a boring fail-open scorer
Eligible prose is scored for keep-probability (heuristic uniform scorer for the no-ML skeleton, or a deterministic token scorer such as an LLMLingua-2-style ONNX path when enabled). Selection is a keep-mask: order-preserving, forced-keeps honored, ties broken by position. Deterministic.
If scoring fails, you don’t get a creative partial answer. You fail open toward safe behavior.
5. A pricing-aware cost gate
Compression isn’t free. It has CPU cost and, under some cache regimes, rewrite cost. The engine estimates whether applying compression is cost-positive given provider pricing and cache model (Anthropic-style cache write multipliers vs implicit OpenAI/Gemini-style caching, etc.). If it doesn’t clear the gate, skip it.
Net: we optimize for dollars and latency under real cache physics, not for a vanity compression ratio on a blog chart.
6. Extra cheap passes that still matter
Alongside extractive keep/delete:
- Whitespace normalization on eligible prose spans
- Dedup of byte-identical text buffers inside the frontier (skip trivial short repeats)
Structural waste dies before cleverness is required.
What this looks like on real workloads
Public workload examples from anyray.ai (before → after tokens, approximate savings). More shapes are in the use-case docs.
| Workload | Before | After | Saved |
|---|---|---|---|
| Access logs (compact repetitive entries) | 11,252 | 230 | ~98% |
| Incident debugging | 17,765 | 1,408 | ~92% |
| GitHub issues / comments | 65,694 | 5,118 | ~92% |
| JSON arrays (task-relevant fields) | 12,560 | 2,893 | ~77% |
| Code search hits | 54,174 | 14,761 | ~73% |
| Git diffs | 3,500 | 1,565 | ~55% |
| Codebase exploration | 78,502 | 41,254 | ~47% |
| Pasted screenshot → OCR text (not vision tokens) | 9,420 | 1,507 | ~84% |
| Spreadsheet export (matching rows + header) | 28,640 | 2,864 | ~90% |
| DB schema for text-to-SQL | 19,880 | 2,386 | ~88% |
| Recurring report instructions | 41,300 | 8,260 | ~80% |
| RAG retrieval chunks | 33,150 | 10,940 | ~67% |
| Email filtering JSON | 60,512 | 3,026 | ~95% |
Headline claim on the site: up to ~60% token reduction across mixed production shapes, without a quality cliff when the gate and protections do their job. Your mileage is workload-shaped — logs and email dumps compress differently than tight code review.
How this is different from “just use LLMLingua / Portkey / LiteLLM”
Short, honest framing. A measured side-by-side is still to come:
- LLMLingua-class compressors are strong building blocks for importance scoring. Anyray can use that class of scorer inside a gateway that also owns routing, fail-open, spend attribution, and cache-stable conversation policy. The compressor is not the product; the product is the pre-model control plane.
- LiteLLM / Portkey-class gateways are excellent at unifying providers, keys, and observability. Anyray’s differentiation is just-in-time structural optimization on the request path, self-hosted in your VPC, with attribution tied to real work — not only proxy + dashboard.
We will publish side-by-side compression ratios, quality checks, and cache behavior. Until that benchmark lands, treat the table above as workload evidence, not a competitor leaderboard. The public comparison we have today is Anyray vs. other token-reduction tools.
Connect vs gateway
Anyray Connect is the on-ramp: a small CLI that points local coding tools at the gateway with a personal key and attribution binding. It configures routing; it does not become a second brain. Provider keys stay server-side on the gateway. Connect doesn’t need to read or persist prompt content to do its job.
The gateway remains where optimization, spend store, and policy live.
Security and operational posture
- Deploy as a container in your account.
- Talk only to model APIs you already use.
- Content stays on infrastructure you trust.
- Optimization modes are explicit; shadow before live.
- Absolute fail-open on optimizer errors.
- Admin visibility: tokens and dollars by user, team, model, project, intent.
If Anyray is down, the correct behavior is boring: traffic passes, models still answer, you investigate later.
Closing
Inference cost isn’t mysterious. A huge fraction of it is structural waste on the path to the model.
Anyray’s bet:
- sit before the model
- remove waste deterministically
- respect prompt cache
- measure in dollars, not vibes
- stay self-hosted and fail-open
If you’re drowning in agent context and your bill looks like a second payroll, that’s the problem we’re building for.
Where to go next
- Get a demoPoint one URL at the gateway and see it on your traffic.
- LLM cost calculatorEstimate annual inference spend and what a self-hosted gateway changes.
- Anyray vs. alternativesHow Anyray compares to other token-reduction tools.
- GatewayWhere Anyray sits on the request path.
- OptimizerHow just-in-time optimization runs.
- Use casesThe workloads the gateway is built for.