Why Compressing Prompts Can Raise Costs 2.7× – Insights from the Caveman Token Trap Paper

Overview

The open‑source Caveman plugin, which gained rapid popularity on GitHub (54 K stars) and Hacker News, advertises a 65% reduction in output tokens by stripping polite filler from model responses. Independent measurements, however, reveal that in realistic coding conversations the overall token savings are only 4‑10%.

Popularity and Claims

Caveman’s core idea is to prepend a system prompt that tells the model to "talk like a caveman" – i.e., delete pleasantries, conjunctions, and any verbose phrasing. The plugin offers several compression levels (Lite, Full, Ultra) and even a "Wenyan" mode that translates output to classical Chinese.

Cost Pressure in Agentic Workflows

In Agentic AI workflows, the bulk of token consumption comes from code generation, file reading, and context understanding, not from polite language. Uber’s experience, reported by the Financial Times, shows that AI‑driven development tools exhausted a year’s budget in just four months, prompting a $1,500 per‑tool monthly cap for employees.

Benchmark Findings

Output compression on most APIs yields a 1.4‑2.4× reduction in actual cost, with best‑case gains of up to 3×.

Input compression triggers the model to generate longer replies as compensation, raising net costs by 1.15× on average and up to 2.7× under the strongest compression.

YapBench (arXiv:2601.00624) shows that models vary widely in "excess output length," with some models producing an order‑of‑magnitude more tokens than necessary.

Limitations and Applicability

For coding tasks, the session‑level token savings remain modest (4‑10%). Over‑compressing prompts can degrade accuracy and cause critical information loss in complex refactoring or configuration changes. In creative or educational scenarios, natural language remains essential; excessive compression harms communication effectiveness.

Model Vendor Responses

Model providers are integrating controllable verbosity parameters (e.g., Claude Opus 4.5’s Verbosity low/medium/high) to let users balance brevity and cost. GitHub Copilot’s shift to usage‑based pricing with AI Credits reflects the same trend: each extra token spoken now has a direct price tag.

Trend Judgment

AI interaction is diverging into distinct use‑case clusters. Tool‑oriented scenarios (coding, automation) favor concise, command‑like output, while collaborative or tutoring contexts require richer, more natural language. The “less is more” mantra is becoming a nuanced, scenario‑dependent decision rather than a universal rule.

Conclusion

The Caveman experiment demonstrates that token‑level politeness is a real cost factor, but the primary savings lie in optimizing the heavy‑weight steps of Agentic workflows. Users should apply prompt compression judiciously, focusing on high‑cost operations and leveraging native model controls rather than relying on blanket prompt trimming.