Model routing sounds like free money. Send the hard work to a frontier model, send the easy work to a small one, and pay less for the same result.

On single prompts, it works. On agent sessions, it often raises the bill, because every switch throws away the prompt cache.

The cache belongs to one model

An agent resends its whole history on every turn. Providers make this affordable with a prompt cache: when a request starts with the same bytes as the last one, that prefix is billed at a fraction of the input price. On Anthropic, a cache read costs 0.1× the base input price and a 5-minute cache write costs 1.25× (Anthropic pricing). OpenAI gives up to 90% off cached input (OpenAI prompt caching).

The cache is keyed to the model and to the provider that served it. A new model has never seen the prefix, so it pays a full cache write on the entire history. The same is true when only the provider changes: Claude on the Anthropic API and the same Claude on Bedrock keep separate caches.

Figure 1. One session over three turns. Illustrative. The small model has no cache for this session, so turn 2 writes the whole history. Turn 3 then writes what the small model added.

The math of one switch

Take a coding agent late in a session: 100,000 tokens of history and 4,000 new tokens per turn. Use illustrative prices of $5 per million input tokens for the frontier model and $1 for the small one. We count input only.

  • Stay on the frontier model. Each turn costs $0.075.
  • Switch one turn to the small model. The turn costs $0.13. The model is five times cheaper, but all 104,000 tokens are now a cache write, so the turn costs 1.7× more.
  • Switch back within five minutes. The round trip costs $0.24, against $0.15 for two turns that stay.
  • Switch back after the cache expired. The round trip costs $0.81, which is 5.4× more.
Figure 2. Input cost of two turns at the end of a long session. Illustrative prices: $5 and $1 per million input tokens. A cheaper model for one turn makes both turns more expensive.

A switch can still pay off. You pay the rebuild once, and you earn it back on each later turn that stays on the cheaper model. The price gap decides how many turns that takes.

Price gapExtra cost of the switch turnSaving per later turnTurns to break even
5× ($5 to $1)$0.055$0.060About 1
2× ($2 to $1)$0.100$0.015About 7
1.25× ($1.25 to $1)$0.111$0.004About 30
Same model, other provider$0.575$0Never

A one-way switch at a session boundary can pay. A switch back and forth does not.

What a smart router sees, and what it misses

A third-party “smart” router scores each request and sends it to the cheapest model that should handle it, often at another vendor. The research behind it is real: in RouteLLM, a learned router kept 95% of GPT-4’s quality on MT Bench and cut cost by over 85%.

But those results come from short prompts, routed one at a time. In an agent session, the router scores one message, and you pay for the whole history.

Figure 3. The same request, as the router sees it and as the provider bills it. Illustrative. The last message decides the route, but the 100,000 tokens of history that come with it decide the cost.
  • It scores the last message. “Rename this variable” looks easy, but it arrives with 100,000 tokens of context.
  • It prices tokens at the list price. Most of an agent request is a cache read at a tenth of that price, so the router sees savings that are not there.
  • It decides each request on its own. Easy and hard turns alternate in a session, and each flip pays the round trip above.
  • A cross-vendor switch resets more than the cache. The tokenizer, the tool-call format and the author of the plan all change.
  • It adds a party to the data path. The router sees every prompt, including your code.

Route sessions, not requests

Routing still saves money in agent workloads when it follows five rules.

  • Decide at the start, then hold. Choose the model when the session starts, and keep it while the cache builds up.
  • Switch at a boundary. Give a step that needs another model its own context: a subagent with a short, fresh prompt. The main session keeps its cache.
  • Keep the model and the provider together. Use fallback for outages, not to spread load.
  • Put the rebuild in the price. Before a switch, compare the rebuild with the saving over the turns that are left.
  • Keep the rest of the prefix still. A change to the tools or the system prompt costs the same rebuild.

What to measure

  • Cache read share per session. If it drops when routing is on, the router is breaking your cache. cacheLint shows where the prefix changed.
  • Cache writes after a route change. A large write right after a switch is the rebuild you paid for.
  • Switches per session. More than one or two means the router follows requests, not tasks.

Then compare the session bill for the same task, with and without routing. That is the only number that tells you what routing saved.

Where to go next

← Back to blog