Scaling Coinbase’s Platform for Spikes in Customer Demand: Lessons, Monitoring, and Traffic Replay

Since 2017, global interest in cryptocurrency has caused Coinbase’s traffic to surge from a stable 25,000 API requests per minute to well beyond its 100,000‑request red line, exposing reliability and scalability challenges that the engineering team needed to address.

During the 2017 peak, service outages occurred when traffic exceeded the red line; the team responded by vertically scaling, upgrading MongoDB versions, optimizing indexes, and splitting hot collections, but continued growth required deeper investigation.

Enhanced monitoring revealed extreme latency (up to 100×) and a mismatch between Ruby server processing times and MongoDB transaction times, leading to “ghost” queries. To diagnose, the team instrumented the MongoDB driver to log queries exceeding latency thresholds, capturing request/response sizes, source code lines, and other metadata, which fed a detailed dashboard.

The dashboard showed a large number of queries generated during user login and dashboard view, caused by a many‑to‑many user‑device relationship and a poor device‑fingerprint algorithm that grouped many users under a single device. Refactoring this to a one‑to‑many relationship (each device maps to a single user) yielded a dramatic performance boost.

Another bottleneck was heavy reads from hot collections. The team introduced a memcached layer that cached query results before hitting the database and invalidated the cache on writes, placing the cache at the ORM/driver level to keep it decoupled from business logic.

These changes proved effective during the traffic spikes of December and January, allowing Coinbase to handle larger loads with improved stability.

To prepare for future spikes, Coinbase built a traffic capture and replay system. The capture tool (a wrapper around mongoreplay) snapshots traffic directed at a chosen MongoDB cluster, encrypts it, and stores it in S3. The cannon tool replays the captured traffic to a newly launched cluster, supporting configurable replay speed and a 10 MB buffer to merge streams from multiple application servers.

Early use of capture and cannon uncovered that the Ruby MongoDB driver interleaved ping commands with find operations, violating driver specifications and contributing to the earlier ghost‑query behavior.

Overall, the engineering effort demonstrates that, alongside security, reliability is critical for Coinbase’s platform to support customers buying, selling, and managing cryptocurrency during periods of extreme demand.