The Token Savings Illusion
Same tasks. Same agents. No optimizers. Up to 54.5% cost difference. We tested 15 AI coding optimizers to find out which savings were real—and which disappeared on a second run.
optimizer setups produced a saving that held up when tested again
how far the bill moved between identical runs with no optimizer at all
early wins that didn't survive a second run
proven savings on the second project we tested
Why we ran this
AI coding agents are billed for every piece of text they read and write. A fast-growing set of optimizers promises to cut that bill, often by 20 to 60 percent. Most of those figures come from the makers, measured once. We measured each optimizer against the same agent doing the same real bug fixes without it, then repeated the runs that looked promising. No maker sponsored the study or saw the results in advance.
Run the same job twice, and the bill can move by 54.5%.
Before testing any optimizer, we ran each agent on the same six bug fixes several times with nothing installed. Same code, same instructions, same agent. The cost still moved.
How to read it
Each dot is one run, placed by how far its cost landed from the typical run. The band spans the cheapest to the most expensive run, and the figure on the right is the gap between them.
Why it matters
This swing is the bar every optimizer has to clear. A one-off comparison showing a 20% saving can sit entirely inside it. Each time we added another run, the swing widened or held steady. It never narrowed.
Four of five early wins didn't survive a second run.
We took the five single-run savings that cleared the noise by the thinnest margins and ran each one again. The hollow dot is what one test showed. The solid dot is the result after two.
How to read it
Each arrow runs from the first result to the result after two runs. Left of zero is cheaper. The shaded band is the normal swing for that agent and project; a dot inside it cannot be told apart from luck.
Why it matters
The size of a first-run saving said nothing about whether it would last. The one that held had the thinnest margin of the five. Treat any single comparison, a maker's included, as a lead rather than a result.
Only 9 of 79 setups produced a saving that held up.
Each bar is one optimizer's effect on the bill, compared with the same agent working without it. Choose an agent and a project.
How to read it
Bars left of zero mean a lower bill. The shaded band is the normal swing with no optimizer; a bar ending inside it could be luck. Dots show each run behind a bar. A saving is proven when it was tested at least twice and lands outside the band.
Why it matters
Most bars are grey, and every proven saving comes from one project. The bars are sorted for readability, not ranked: the proven savings sit closer together than the normal swing, so this data cannot say which optimizer is best.
Real savings come from two different places.
An agent's bill is the number of steps it takes multiplied by the cost of each step. For every proven saving, this shows which of the two moved.
How to read it
Further left, the agent took fewer steps. Further down, each step was cheaper. The curved lines join points with the same total saving: 0%, 25% and 50%.
Why it matters
A fewer-steps optimizer changes how the agent works through a problem. A cheaper-steps optimizer trims what it reads. They solve different problems, so one may suit your codebase when the other does not.
The same optimizer can save money with one agent and cost more with another.
Every result in the study: 15 optimizers, three agents, two projects.
Show every result as a table
How to read it
Each row is an optimizer and each panel an agent. Dots left of the centre line lowered the bill; dots to the right raised it. A missing dot means there is no working version for that agent.
Why it matters
Dots for the same optimizer often land on both sides of the line. Which agent you run can move the bill as much as which optimizer you install.
A 30% saving in one run isn't a result. It's a reason to run the test again.
Before you buy an optimizer, ask three questions.
How many runs is the claim based on?
One before-and-after comparison can show a 20 to 50 percent "saving" by chance. Ask for repeated runs and the spread between them.
Was it measured on code like yours?
Every proven saving here came from one project. None carried over to the second. Run a trial on your own repository.
Was it measured with your agent?
The same optimizer lowered the bill with one agent and raised it with another. Test with the agent and model you actually use.
What this study cannot tell you
- Which optimizer is best. The proven savings sit closer together than the normal swing.
- That unproven means useless. — of — setups were tested only once, which is not enough to decide either way.
- Anything from run counts. Promising first results were re-tested, so more runs does not mean better.
- Agent versus model. Claude Code runs Claude Opus 5; Codex CLI and OpenCode run GPT-5.6 Sol, so the two are tied together.
- Quality. Cost was counted whether or not a fix was right. — of — fixes passed the project's own tests.
How we tested
- Real work. Six genuine bugs per project, taken from each project's history and checked by its own tests.
- Identical instructions. The agent is told the symptom only, never where to look and never to use the optimizer.
- Fair installation. Each optimizer installed the way its maker recommends, tested on its own.
- Frozen versions. Agent versions were pinned for the whole study.
Terms
- AI coding agent
- Software such as Claude Code, Codex CLI or OpenCode that reads a codebase and changes it on its own.
- Optimizer
- An add-on that promises to lower what the agent is billed for.
- Token cost
new text + 0.1 × re-read text + 6 × written text, mirroring how AI providers price usage.- Proven saving
- Tested at least twice and cheaper by more than the normal swing with no optimizer.
Data and code
- Every figure on this page is calculated from the published raw data, not typed by hand.
- Every retained run is published, including failed fixes and expensive outliers.