How Coinbase Halved AI Costs While Token Usage Continued to Surge

Coinbase AI Cost‑Optimization Overview

Coinbase publicly shared a three‑year internal plan that reduced AI dollar spend by nearly 50% while token usage continued exponential growth. A chart (shown below) compares total token consumption (rising sharply) with departmental AI spend (flat or declining after the measures).

Five concrete actions

Default model configuration optimization – Engineers may select any model, but the internal LLM gateway defaults to cost‑effective open‑source models (e.g., GLM 5.2, Kimi 2.7). Expensive frontier models are invoked only when a task requires them. Code reviews include cross‑model result verification to guard quality.

Intelligent routing – A custom workflow pre‑processes prompts, then automatically assigns the most suitable model based on task type, cache‑hit status, and per‑model pricing. Complex planning tasks use frontier models; simple execution steps use cheaper alternatives, eliminating manual model selection.

Cache‑reuse enhancement – All large‑model requests are made cache‑aware. On the internal LibreChat tool, cache‑hit rates rose from 5 % to 60 % after the change.

Context minimization – New sessions are started for each distinct task, and only necessary file context is retained. The goal is to avoid wasteful token consumption rather than merely compress tokens.

Transparent usage visualization – Each engineer’s token consumption and model choice are publicly displayed, linking higher AI budgets to higher expected output.

Box CEO Aaron Levie noted that while the tactics are practical details, successful rollout requires deep, non‑abstract understanding of specific business workflows; merely tweaking model parameters is insufficient.

Implications for the AI‑stack market

The discussion highlighted a broader opportunity: a middle‑layer that adapts large models to concrete enterprise workflows. Building such a layer at scale is difficult for individual companies, creating space for specialized providers. The layer’s value derives from scenario‑specific model evaluation, vertical knowledge integration, and product‑level adaptations rather than the raw model itself.

Community feedback and risk examples

A developer posted a “meaningless metrics museum” illustration, criticizing token count as a vanity KPI when it does not correlate with actual value.

Another developer recounted an overnight Agent cluster that ran without routing rules, consuming expensive models and burning a large budget.

Some participants argued that intelligent routing is hard because each model has distinct command specifications, while others predicted convergence of model interfaces over time.

Investor perspective

Investor USV partner Nick Grossman’s AI‑stack layering article was cited, comparing profit capture across layers to the Android‑Apple split. The analysis suggests that identity and settlement layers will concentrate most profits, while other layers may see margin compression.

Related product responses

Redis announced Langcache , a prompt‑caching tool for agents that addresses cache‑reuse challenges.

Open‑source LLM‑gateway creators expressed willingness to support Coinbase’s approach.

Open‑source personal‑grade solution

The project Token Bank (https://github.com/wink-run/local-llm-proxy) implements a personal‑grade LLM gateway mirroring Coinbase’s scheme: intelligent routing, cache compression, usage visualization, and optional peer‑to‑peer compute sharing. It can be placed between AI tools (e.g., Claude Code, Cursor, Codex CLI) and various model providers.

Many companies remain in a wild‑growth phase of AI spending, either imposing hard caps or burning money without ROI insight. Coinbase’s proven, practical scheme provides a reference for cost‑conscious enterprises and entrepreneurs building AI middle‑layer solutions.