Claude Code vs. Hermes & Kimi Code: Up to 30× Token Consumption Difference

Composio tested the same large‑language model (Kimi K3) inside three different agent harnesses—Claude Code, Hermes, and Kimi Code—on 28 identical tasks to measure success rate, token consumption, cost, and speed.

The success rates were close: Kimi Code succeeded on 22/28 tasks, Hermes on 21, and Claude Code on 20. The main divergence was token usage. Median token counts were 61 k for Kimi Code, 67 k for Hermes, and a striking 340 k for Claude Code—about six times more than Kimi Code.

Using the pricing of $3 per million input tokens for Kimi K3, the average cost per task was calculated as $0.22 for Kimi Code, $0.28 for Hermes, and $2.00 for Claude Code, highlighting a clear cost gap.

Latency also varied: median runtimes were 179 s (Hermes), 297 s (Kimi Code), and 348 s (Claude Code). Thus, the fastest harness was Hermes, while the most token‑efficient was Kimi Code; the two advantages do not coincide.

Composio concluded that choosing the right harness can reduce agent costs far more than swapping models, as the harness alone created a nine‑fold cost difference while model performance remained comparable.

Sebastian Raschka echoed these findings, noting that Claude Code often consumes two to three times more tokens than other harnesses despite similar success rates.

Raschka investigated why Claude Code uses more tokens and found the excess lies mainly in input tokens. In one run Claude consumed 578 k input tokens but produced only ~4.5 k output tokens over 25 interaction rounds, suggesting the harness repeatedly feeds extensive context (previous messages, tool calls, command outputs, file contents) back to the model.

The broader implication is that harness design is as critical as the underlying model.

A recent paper from Writer (an enterprise AI‑agent platform) performed a controlled‑variable experiment on 22 enterprise tasks using six base models (Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6). By swapping only the orchestration layer to Writer’s own harness, they observed average cost reduction of 41 % (from $0.21 to $0.12), median latency reduction of 44 % (48 s to 27 s), token consumption reduction of 38 % (14.2 k to 8.8 k), while task quality remained statistically unchanged (0.78 → 0.81).

These results suggest that, once model performance is a given, the harness becomes the decisive factor for cost efficiency—essentially the “harness tax” that should be accounted for in benchmarks.

In future agent competitions, the first half will test capability (“can it do it?”) and the second half will test efficiency (“who does it cheapest?”), with the latter hinging on harness design rather than model choice.

Therefore, accounting for the “harness tax” and optimizing orchestration layers are essential steps toward more economical AI‑agent deployments.