Evaluation-driven compression

We study how to remove context from agent requests without breaking the prompt cache, the agent or the answer. Then we measure it in dollars, in the open.

The compression engine,as a research problem

Removing tokens is not the hard part. The hard part is removing them so the session bill goes down and the agent still works.

Same input, same bytes

Compression is a pure function of each message. The provider sees the same bytes on every turn, so the cached prefix keeps matching.

Priced, not guessed

Each edit is priced against the provider’s cache rules. An edit ships only when it lowers the bill over the rest of the session.

Removed is not lost

Removed content stays one tool call away, byte for byte. Most lossy edits become deferred reads.

Learned per tool, per team

A decision engine picks edits per request, checks them against your own traffic and backs off when the model reacts badly.

Benchmarked on real-world tasks

Terminal-Bench 2 tasks on Claude Sonnet 5.5, graded by the Harbor task verifier.

Each solved task costs 38% less

We take the cost of every trial and divide it by the trials that passed. One solved task cost $0.228 with Anyray and $0.367 with Claude direct.

The full bill is 32% lower

This is every trial added up, passed or failed, as a share of what the same trials cost with Claude direct. Anyray cost 68% of that, and it cost less on 95% of the tasks.

The same work finishes 18% sooner

This is the time for 50 tasks, from the average time per task. Anyray took 6h 08m and Claude direct took 7h 30m. The gap follows the step count further down: fewer model calls, less waiting.

The agent needs 24% fewer steps

A step is one model call, and every call sends the session again. Fewer steps means less to pay for and less to wait for. The agent finished a task in 8.7 steps with Anyray and 11.4 with Claude direct.

50% fewer cache busts

A cache bust is a request that changes the start of the session. The provider then bills the whole context again at the full input price, instead of the cheap cache rate. Anyray had 8 per 100 trials. Claude direct had 16.

The numbers,per workload.

Evaluation-driven compression across workloads

By function

  • Data

    38%

  • Coding

    30%

  • Tools & MCPs

    68%

  • Memory & recall

    79%

By workload

Access logcompacting repetitive entries down to the relevant few96%
Incident debuggingkeeping only the signals that point to the cause84%
Synonym-gap logsfinding an OOM event the question never names61%
GitHub issuesreading GitHub issues/comments to categorize/prioritize84%
JSON arraykeeping only the fields and items the task uses42%
Code searchgrep/semantic search returning many file+line hits56%
Git diff outputreviewing what changed in a commit or PR65%
Test-suite outputkeeping only the failed tests and the lines that explain them77%
Flaky-suite re-runcollapsing a near-duplicate test re-run to what changed37%
Codebase explorationreading source files to understand architecture28%
Read a Python modulekeeping only the function bodies on the task path70%
Read a large TS filekeeping only the code paths a TypeScript service task touches66%
Multi-file tracefollowing one call path across several files33%
Recurring reportremoving repeated instructions across a nightly batch80%
RAG retrievalkeeping only the chunks that answer the question68%
MCP tool bloatkeeping only the tool schemas the task calls70%

Run the benchmark yourself.

The harness, the scenarios and every result are open source. Point it at your own key and compare the two arms on your own tasks.

anyrayHQ/benchmarks

Selected reading

Research

The decision engine: how Anyray learns what is safe to cut

Research

Why context compression breaks prompt caching, and how to keep the prefix stable

Research

How retrieval handles make lossy compression safe

Engineering

cacheLint: detect prompt-cache breaks in LLM agents

Ready to run AI without the waste?