Meituan's Kafka Optimizations: Challenges, Latency Improvements, and Large‑Scale Cluster Management

1. Current State and Challenges

Meituan's data platform uses Kafka as a unified streaming storage and distribution layer, handling more than 15,000 broker machines, clusters up to 2,000 nodes, and daily traffic exceeding 30 PB with peak rates over 400 M messages per second. As the cluster grows, several challenges emerge:

Slow nodes (TP99 > 300 ms) caused by load imbalance, PageCache pressure, and consumer thread model flaws.

Complexity of large‑scale cluster management, including topic interference, insufficient broker metrics, and delayed fault detection.

2. Read/Write Latency Optimizations

Optimizations are divided into application‑layer and system‑layer improvements.

Application layer :

Disk balance via a priority‑based partition migration plan managed by a Rebalancer component.

Migration acceleration using pipelined commits, migration cancellation to avoid PageCache pollution, and Fetcher isolation to separate real‑time and lagging reads.

Consumer asynchronous processing by introducing background fetch threads to avoid single‑thread bottlenecks.

System layer :

RAID card acceleration to mitigate HDD random‑write latency.

Cgroup isolation to prevent resource contention between IO‑intensive Kafka and CPU‑intensive Flink/Storm workloads, and to keep Kafka on a single NUMA node.

SSD‑based hybrid cache architecture that stores recent segments on SSD, synchronizes to HDD, and avoids PageCache pollution.

3. Large‑Scale Cluster Management Optimizations

Isolation strategies across business domains, roles (Broker vs. Controller), and priority tiers (VIP clusters).

Full‑link monitoring that collects metrics from all Kafka components, enabling rapid pinpointing of latency hotspots.

Service lifecycle management that ties service state to machine state, automating status changes and preventing manual errors.

TOR disaster recovery to ensure replicas of a partition are placed on different racks, guaranteeing availability during rack failures.

4. Future Outlook

Future work will focus on enhancing robustness through finer‑grained isolation, client‑side fault avoidance, hot‑swap capabilities, and exploring cloud‑native deployments of the streaming service while maintaining cost efficiency.