How Cloudflare Scaled Kafka to Process Over 1 Trillion Messages: Ops Lessons and Architecture

Cloudflare Overview

Cloudflare operates a global network that secures websites, APIs, and traffic, and enables edge deployment of applications. Its infrastructure consists of a global edge network and a control plane built on Kubernetes, Kafka, and databases, with Workers running at the edge.

Kafka

Kafka clusters are composed of multiple brokers with a leader broker coordinating communication. Topics are partitioned for horizontal scalability, each partition having a leader and replication factor (minimum 3). Producers send messages, consumers read them.

Engineering Culture

Cloudflare moved from a monolithic PHP stack to a flexible, language‑agnostic approach, encouraging teams to choose their own tools while providing shared services and best‑practice libraries.

Decoupling via a Generic Message Bus

To reduce tight coupling, a generic message‑bus cluster was created, allowing teams to publish to new topics with predefined replication, retention, and ACLs, enabling independent evolution of services.

Unstructured Communication

Initially, teams used arbitrary JSON messages, leading to incompatibilities. The organization adopted protobuf for strict schemas, forward/backward compatibility, and multi‑language code generation, using Uber’s Prototool for style enforcement.

Connectors Framework

A connector framework built on Kafka Connect lets engineers create services that read from one system and push to another (e.g., Kafka to Quicksilver). Configuration is driven by environment variables, and custom transformers are the only required code.

Visibility

During rapid growth, audit‑log pipelines exposed scalability challenges. Metrics were added to the SDK, and Prometheus histograms helped identify slow steps. OpenTelemetry was evaluated for tracing, but limited Kafka integration required custom health checks.

Alert Noise

High metric volume generated noisy alerts. The alert pipeline uses Prometheus, Alertmanager, and PagerDuty, with Kubernetes health checks (liveness, readiness, startup) adapted for Kafka services.

Consumer Lag and Batch Processing

Consumer lag caused by production spikes was mitigated by switching to batch processing, handling a fixed number of messages at a time, which improved throughput and reduced latency.

Documentation

To aid SDK users, a Google Chat channel and internal wiki capture common issues, bugs, and answers, improving the overall developer experience.

Conclusion

Balance flexibility and simplicity in configuration.

Add visibility early with metrics.

Enforce strong contracts for message schemas.

Document solutions to reduce support overhead.