How LinkedIn Scales Kafka to Billions of Messages Every Day

If data is the lifeline of high‑tech, Apache Kafka is the circulatory system that powers LinkedIn’s massive data movement across dozens of systems, handling billions of messages each day.

What is Kafka?

Apache Kafka is an evolved publish/subscribe messaging system that combines queue and log semantics. Messages are organized into topics and partitions, supporting multiple producers and consumers. Each Kafka cluster retains messages reliably and efficiently.

Time‑based retention (e.g., LinkedIn measures over days)

Size‑based segment retention

Key‑based retention keeping only the latest value

Kafka offers reliability, flexibility, and high throughput.

How big is “big”?

LinkedIn runs over 60 clusters with more than 1,100 Kafka brokers, processing over 8 trillion messages daily (≈175 PB inbound, 650 PB outbound). At peak, it ingests >13 million messages per second (≈2.75 GB/s).

Message Types

Queues : Standard messages for coordination and state, used for email, data distribution, and backend integration.

Metrics : System and hardware statistics that drive internal monitoring and alerting.

Logs : Application, system, and access logs that were originally co‑located with metrics but are now separated due to volume.

Traces : Detailed records of infrastructure actions, feeding stream processors like Apache Samza and batch jobs in Hadoop for indexing, usage tracking, and real‑time analytics.

Layered Architecture and Aggregation

LinkedIn’s data centers host local message containers that aggregate into a global cluster via Kafka MirrorMaker, reducing cross‑data‑center traffic and latency. The hierarchy consists of producers (first layer), local clusters (second layer), and aggregate clusters (additional layers), with consumers at the top.

This layering improves bandwidth usage but adds monitoring complexity; each layer must be audited to ensure no message loss.

Audit Integrity

LinkedIn’s internal Kafka Audit tool adds a header with timestamp, producer service, and host to every message. Producers periodically publish counts to audit topics, allowing comparison with consumer metrics to detect missing or duplicated messages.

Kafka Console Auditor consumes all topics, aggregates counts, and verifies that each layer sees the same volume, guaranteeing end‑to‑end delivery.

Putting It All Together

LinkedIn’s dedicated Kafka engineering team, comprising core open‑source contributors, provides internal libraries (e.g., Tracker Producer) that enrich messages with headers and audit data, and maintains custom tools like the console auditor.

Outlook

LinkedIn continues to push Kafka’s limits, targeting daily volumes of 10 trillion messages, enhancing security, quota controls, and integrating with the Samza stream‑processing framework. The SRE team automates operational tasks such as partition rebalancing, while the company remains active in the Apache Kafka community through meetups and open‑source contributions.