LinkedIn, the original home of Apache Kafka, abandoned the platform for a home‑grown system called Northguard, redesigning log storage, decentralizing metadata, and adding a virtualized Xinfra layer to handle trillions of daily events, while still acknowledging Kafka’s relevance for most companies.
LinkedIn, facing planetary‑scale data volumes, found Kafka’s architecture hitting fundamental limits and built Northguard—a decentralized, log‑striped streaming platform with Raft‑based metadata and an Xinfra migration layer—to gradually replace Kafka’s core responsibilities while maintaining compatibility.
LinkedIn, the creator of Kafka, has largely abandoned Kafka in favor of a new log storage system called Northguard, whose design mirrors Apache Pulsar with features like storage‑compute separation, log striping, and a multi‑layer data model, offering superior scalability, operability, consistency, and durability for massive data streams.
