Token reduction alone is insufficient evidence of a lower bill. A tool can remove tokens, break the work and raise the cost.

The fair test is the same work, with and without the tool. We ran Claude Code on Terminal-Bench 2 tasks, direct and through Anyray. Through Anyray, the agent solved 82% of the tasks against 76% direct, and the total cost was 32% lower. We ran RTK and Headroom on the same tasks. RTK raised the total cost by 2%, and Headroom by 40%.

This post gives the setup, the tasks and the numbers. It also explains what an agent pays for, because that is the reason why some tools remove tokens and still raise the cost.

What we compared

An arm is one configuration of the test. Four arms ran the same tasks. The agent and the model stayed the same. The only change was the tool between the agent and the model.

  • Direct. Claude Code talks to the model with nothing in between. This is the baseline.
  • Anyray. A compression gateway that reduces costs across context, tools, files, dev commands, codebase and cache.
  • RTK v1.2.0. A command-line tool on the developer's machine. It trims the output of shell commands before the agent reads it.
  • Headroom v0.44.0. An open-source compression for AI agents.

The setup

  • Benchmark. Terminal-Bench 2, a public benchmark of real terminal work for coding agents.
  • Agent. Claude Code 2.1.322 on Claude Sonnet 5.5, at medium effort.
  • Grading. Harbor 0.25 runs the verifier of each task. A trial passes or fails.
  • Trials. A trial is one run of one task. Each task ran more than once in each arm.
  • Cache. Each trial started with a cold prompt cache.
  • Cost. The dollars billed for a trial, as Harbor reports them.

Harbor is the harness of the benchmark. It starts a clean container for each trial, runs the agent, and then runs the verifier. The agent cannot see the tests. This is important for a compression test: a tool cannot look good because it made the answer shorter. The work must pass.

The tasks

A Terminal-Bench task gives the agent a container, a written instruction and a time limit. The agent works in the terminal: it reads files, runs commands, writes code and reads the output. When it stops, a verifier runs tests against the container.

The tasks cover building software, debugging, system administration, security and scientific computing. We ran a mix of two sizes on purpose.

  • Short tasks finish in 2 to 6 minutes and take a small number of turns. They show what a tool costs when there is little history to compress.
  • Long tasks can run for more than 30 minutes. The agent resends a growing history on every turn, so they show what a tool does to a long session.
Short taskLong task
Recover lost commits and merge them into the main branch.Download, patch and compile an old ray tracer from source.
Build a gRPC key-value server and leave it running.Fix a garbage collector bug so the OCaml compiler builds itself again.
Make an RSA key and a self-signed certificate, then check them.Write a small C program that renders a given image.
Find a security bug in a web framework, report it and fix it.Design a circuit of logic gates that computes a Fibonacci number.

An engineering team has both sizes in one day. A tool that helps on only one size does not lower the monthly cost by much.

What an agent pays for

The results are easier to read when you know where the money goes. A model API keeps no memory between calls. So on each turn, the agent sends everything again: the system prompt, the tool list, each earlier turn and each tool output. The provider caches the start of the request and bills it at a lower price when the next request starts with the same bytes.

Published measurements of Claude Code sessions show the result. About 80% of the cost is cache traffic, which is cache reads and cache writes. Model output is about 10%. Fresh input, which is the new text that is not yet in the cache, is about 1%.

The text that a tool-output compressor can change is a small part of this. The same measurements put it at about 5% of the cost. Four mechanics follow, and they explain each result in this post.

  1. Most of the cost is cache traffic. The agent sends the full history again on each turn. The provider bills most of it as cache reads and cache writes.
  2. A removed token saves little. A cache read costs 0.1x the input price. A token that is in the cache is already cheap, so its removal saves a small amount.
  3. An extra turn costs a lot. If a tool removes text that the agent needs, the agent searches or reads again. Each extra turn sends the full history again.
  4. A changed prefix is expensive. A cache write costs 1.25x the input price. A tool that changes the cached part of a request turns cache reads into cache writes.

So the token count is the wrong target. A tool lowers the cost when it lowers the number of turns and keeps the cache stable. The sections below measure these two things directly.

Quality: more tasks solved than direct

The first question for a compression tool is what it breaks. A pass in this benchmark is strict. When the agent stops, the verifier runs tests in the container. The agent cannot see those tests, so a shorter answer does not pass. The work must be correct.

Compression can break the work in four ways.

  • Broken edits. An edit must match the code in the file byte for byte. If a tool rewrote that code, the edit fails.
  • Broken pipes. The agent sends the output of one command into a second command. If a tool changed the output, the second command gives a wrong result.
  • Lost evidence. Test output, tracebacks and logs are dense. A tool that trims them can remove the one line that the agent needs.
  • Extra work. The agent does not fail. It searches and reads again, and the cost goes up.

Direct, the agent passed 76% of its trials, so about one trial in four fails with no tool. That leaves room for the result to move in both directions, and a tool that broke edits, pipes or evidence would push the pass rate down. Through Anyray, it went up to 82%.

Cost: 32% less than direct

Total cost is the cost of all tasks in an arm, compared with the same tasks direct. Anyray lowered the total cost by 32%. RTK and Headroom also removed tokens, but their total cost went up, by 2% and 40%.

The mechanics above explain the order. RTK changes only shell output, which is a small part of the request, so it has little to gain. Headroom changes more of the request, and its cache bust count below shows the price of that.

Total cost does not include quality. A failed trial costs money and gives no result. So we also divide the cost of all trials, failed trials included, by the number of trials that passed.

One solved task cost $0.228 through Anyray and $0.367 direct. That is 38% less than direct, and 27% less than RTK.

Task by task

One large saving can move an average. So we looked at each task alone. Through Anyray, 95% of the tasks cost less than direct. With RTK it was 53%, and with Headroom 11%.

This number tells you how sure the saving is. At 53%, a tool is close to a coin toss: it helps on one task and costs more on the next. At 95%, you can turn the tool on for a full team and not select the sessions.

The savings appear on both task sizes. The best short task cost 42% less than direct. The best long task cost 36% less.

Turns per task

A turn is one call from the agent to the model. Each turn sends the full history again, and the provider bills that history on each call. Late in a long task, one turn can send tens of thousands of tokens.

Through Anyray, the agent finished a task in 8.7 turns on average, against 11.4 direct. That is 24% fewer. Each turn that the agent does not need removes a full request from the cost, not a small part of one, so the turn count has more effect on the cost than the number of tokens removed. We did not measure why the agent needs fewer turns. The data shows only that it does.

Cache busts

A cache bust is a turn that reads less than 90% of what the last turn left in the cache. The provider then writes that part of the prompt again. A cache write costs more than ten times a cache read.

Anyray had half the cache busts of direct. Headroom had three times as many as Anyray, and more than direct. A tool that changes the cached part of a request turns cache reads into cache writes. The token count goes down and the cost goes up. Our post on compression and the prompt cache explains how this happens.

Anyray is below direct because it protects the cached prefix by design. It replays the same edit on each later turn, so the bytes that the provider cached do not change. It edits text that is already in the cache only when the saving for the remaining part of the session is larger than the cost of the new cache write. On a short session it makes no such edit.

Latency: fewer turns, faster tasks

Cost is not the only result. Through Anyray, the agent also finished sooner. Three things make a session faster.

  • Smaller requests. The model reads less text on each turn, so it starts its reply sooner.
  • A warm cache. The provider reads a cached prefix faster than it processes the same text again. Fewer cache busts mean fewer slow turns.
  • Fewer turns. Each turn is a wait, because the model reads the request and writes a reply before the agent can do its next step.

Wall clock

Wall clock is the real time of a full trial, from start to end. It includes the start of the container, the work of the agent and the run of the verifier, so it is the time that a developer waits for a result.

The time of a trial has three parts: the model reads the request and writes a reply, the tools run, and each tool in the path of a request adds its own delay. Fewer turns shorten the first part. Compression itself can cost a model call. A tool that summarizes with a model pays time, and money, on each turn.

Through Anyray, the agent finished 18% sooner than direct. All three reasons apply here. The requests are smaller, the cache bust count is half that of direct, and the agent needed fewer turns: 8.7 for a task against 11.4 direct. We did not measure how much of the 18% comes from each one.

Headroom shows that a tool in the path can add time. It took 9h 21m against 7h 30m direct. Its turn count is the same as direct, at 11.4 turns for a task, so the extra time does not come from more turns. It comes from the tool itself.

Run this test on your own traffic

Terminal-Bench is public, but your traffic is the test that counts. Compare provider charges and task outcomes across representative workloads. Include retries, caching, additional model calls, and any service fees. A useful experiment holds the task and acceptance criteria constant and checks whether the work still succeeds.

Measure four things.

  • Cost per solved task. Divide the full cost, failed trials included, by the number of trials that passed. Do not use tokens removed.
  • Pass rate. A tool that costs less and fails more does not save money.
  • Cache busts. If they go up when the tool is on, the tool is breaking your cached prefix. cacheLint shows where.
  • Turns per task. More turns can mean that the agent reads again what the tool removed.

Where to go next

← Back to blog