Why LinkedIn Dumped Kafka for Its Own ‘Northguard’ Streaming Engine

Origin Story: Kafka Was the Darling

In 2010, LinkedIn faced a data‑deluge with 90 million users and needed a real‑time log and event transport solution, leading to the creation of Apache Kafka, which quickly became a flagship project of the Apache Software Foundation.

By 2026, LinkedIn serves 1.2 billion users and processes over 32 trillion records per day, outgrowing the original Kafka architecture.

Cracks Appear: The Issues

At LinkedIn’s massive scale, Kafka shows several pain points:

Metadata nightmare: Managing 150 clusters and 400 k topics per cluster overloads the central controller, creating a bottleneck.

Rebalancing pain: Adding a broker requires extensive data movement, making scaling slow and risky.

Resource imbalance: Hot and cold partitions cause uneven disk usage, leading to nightly operational headaches.

Northguard Appears: A New King

Instead of patching Kafka, LinkedIn built Northguard, a fundamentally re‑architected streaming system:

Log segmentation: Logs are split into 1 GB chunks, turning a single massive partition into many small, automatically balanced segments.

Decentralized metadata: Metadata is sharded across the cluster using a Raft‑based state machine, eliminating the single‑point‑of‑failure controller.

Xinfra – the migration layer: A virtualized publish/subscribe layer that lets applications talk to both Kafka and Northguard seamlessly, handling 32 trillion daily records without downtime.

Kafka vs. Northguard: The Showdown

For most organizations, Kafka remains the “bread and butter” of streaming, especially managed services like Confluent. LinkedIn’s switch is an outlier driven by its unique scale, akin to needing a Formula 1 car when a reliable SUV would suffice for most.

FAQ: Your Real Thoughts

Can I download Northguard? Not yet; it is internal to LinkedIn and only hinted at being open‑sourced in the future.

Should I stop learning Kafka? Absolutely not. Kafka is still the industry standard for the vast majority of use cases.

How hard is migration? LinkedIn’s Xinfra makes migration appear simple for them, but for most teams the existing DevOps complexity makes a full replacement daunting.

Conclusion

LinkedIn’s replacement of Kafka with Northguard marks a significant engineering milestone, showing that even the most successful tools can hit limits at extreme scale. Whether Northguard will become a widely adopted open‑source project remains to be seen, but the effort underscores the importance of architectural innovation for massive data pipelines.