How we benchmark token tools (methodology)

July 8, 2026 · Tokeasy

Every token tool claims to save you money. Most back it with a number that's easy to game and hard to trust: tokens saved per call. We think that metric is actively misleading, so before publishing any results we want to explain exactly how we measure — including where our own numbers stop.

Why cost-per-task is the only honest metric

The tempting metric is "tokens per tool call." It's easy to measure and it makes any compression tool look great: strip 40% out of each result and you can claim 40% savings. The problem is that an agent is a feedback loop. When you strip information out of a result, the agent frequently notices and re-fetches it — so the tool that scored best on tokens-per-call can be the tool that made the agent do the most extra round-trips.

The only metric that can't be gamed this way is total cost to complete a real task: every token, across every round-trip, from the first prompt to a finished, correct result. If a tool "saves" tokens per call but triggers three extra re-fetches, cost-per-task exposes it. If a tool genuinely reduces the work, cost-per-task rewards it. So that's what we measure, and nothing else goes in the headline.

Suite design

The suite is five real repository tasks × five configurations. The tasks are things an agent is actually asked to do, not synthetic micro-benchmarks:

• Build a VR physics simulation game feature
• Provision a SaaS worker
• Run a security audit over a SaaS codebase
• Do a piece of financial deep-research tooling
• A multi-day mobile-app refactor

Each task is run under five configurations, same model throughout — only the efficiency layer changes:

1. Stock Claude Code — the baseline, no tooling changes.
2. Headroom — a token-stripping proxy.
3. RTK (Rust Token Killer) — a stripping/compression layer.
4. Caveman — another reduction proxy.
5. Tokeasy — our drop-in tools.

What we measure

For every (task, configuration) pair we record the total cost to drive the task to completion — all input tokens (including the transcript re-billed on each turn), all output tokens, across all tool calls and retries. Not the cost of the winning path in hindsight: the actual cost of the run, re-fetches and failed edits included, because that's what you actually pay. A run only counts if it reaches the same finished state as the baseline; a cheaper run that doesn't finish the task isn't cheaper, it's incomplete.

Results

The full per-task breakdown, with animated cost bars for each tool on each task, is on the benchmarks section of the site. The suite-level summary:

• −19.8% total suite cost versus stock Claude Code.
• 3.0× the savings of the next-best tool on the same suite.
• 5 of 5 tasks where Tokeasy had the lowest cost to completion.

The result that surprised people most: the token-stripping proxies generally lost to the plain baseline on cost-per-task. That's not a knock on the engineering in those tools — it's the re-fetch dynamic showing up exactly where the honest metric can see it. Smaller calls, more of them, higher total.

Limitations (the part most benchmarks bury)

We'd rather state these plainly than have you find them:

• It's one suite. Five tasks is enough to show a direction, not enough to be a law of nature. Your repositories and your tasks will differ.
• Results are workload-dependent. The savings come from cutting plumbing. A task that's light on file exploration and heavy on pure reasoning has less plumbing to cut, so it'll show a smaller gap.
• Costs are estimates where they have to be. Our savings ledger labels every number counted, measured, or estimated, and we hold to that distinction in the benchmark too. Where a figure is modeled rather than metered, we treat it as an estimate.
• We built one of the five tools. We ran the suite as fairly as we know how, but we're not a neutral third party, and we won't pretend otherwise.

Reproduce it

The reason we publish the methodology and not just the number is that the methodology is the claim. If you run your own workload through the same cost-per-task lens — baseline versus any tool, measured to completion — we want to hear what you find, including if it disagrees with us. The metric is the point; the percentage is just what our suite happened to produce.

claude mcp add tokeasy -e TOKEASY_LICENSE_KEY=*** -- npx -y tokeasy