Meituan’s Kafka Optimizations: Reducing Read/Write Latency and Managing Large‑Scale Clusters

This article describes how Meituan’s data platform tackles the growing challenges of a 15,000‑plus‑node Kafka deployment by detailing current bottlenecks, latency‑reduction techniques across application and system layers, large‑scale cluster management strategies, and future directions for robustness and cloud‑native migration.

Big DataKafkaLarge-Scale Clusters

0 likes · 21 min read

Meituan’s Kafka Optimizations: Reducing Read/Write Latency and Managing Large‑Scale Clusters

dbaplus Community

Aug 23, 2022 · Big Data

How Meituan Scaled Kafka to 15k Nodes and Overcame Latency & Management Challenges

Meituan’s data platform uses Kafka as a unified cache and distribution layer, and as the cluster grew beyond 15,000 brokers and daily traffic exceeded 30 PB, the team tackled slow nodes, read/write latency, large‑scale cluster management, full‑link monitoring, service lifecycle, and future cloud‑native deployment challenges.

KafkaLarge-Scale ClusterRead‑Write Latency

0 likes · 22 min read

How Meituan Scaled Kafka to 15k Nodes and Overcame Latency & Management Challenges

Meituan Technology Team

Aug 4, 2022 · Big Data

Optimizing Kafka for Large-Scale Data Platforms at Meituan

The article details Meituan's massive Kafka deployment—over 15,000 machines handling more than 30 PB of daily data—its performance and management challenges, and the comprehensive application‑layer, system‑layer, and hybrid‑layer optimizations Meituan implemented to reduce read/write latency and improve large‑scale cluster reliability.

Cluster ManagementData PlatformFull‑Link Monitoring

0 likes · 25 min read

Optimizing Kafka for Large-Scale Data Platforms at Meituan

DataFunTalk

Feb 1, 2022 · Big Data

Kafka at Meituan: Practices, Challenges, and Optimizations for Large‑Scale Data Platforms

This article presents Meituan's large‑scale Kafka deployment, describing the current state and challenges of massive data ingestion, detailing latency‑reduction techniques, cluster‑level optimizations, SSD‑based caching, isolation strategies, full‑link monitoring, lifecycle management, and future directions for high availability.

Cluster ManagementKafkaMeituan

0 likes · 22 min read

Kafka at Meituan: Practices, Challenges, and Optimizations for Large‑Scale Data Platforms