Choosing the Right Disk Strategy for High‑Throughput Kafka Clusters

Background

During the pandemic, Tencent Medical relied on Tencent Cloud CKafka to provide timely epidemic information, requiring massive data ingestion and storage. Kafka’s role as a log‑analysis system demands high throughput (GB/s) and robust storage, prompting a careful choice of underlying disk solutions.

Key Design Factors

When designing disk storage for Kafka clusters, the following factors must be considered:

Large‑capacity storage

High I/O throughput

Rapid scaling (vertical/horizontal)

Data safety

Low redundancy overhead

Common Disk Architectures

Typical Kafka storage setups include single‑disk read/write, multi‑directory read/write, RAID arrays (RAID0, RAID10), and logical volume management (LVM) striping. Each approach has distinct advantages and trade‑offs.

1. Single‑Disk Read/Write

This is the simplest and quickest to deploy, often used in small‑scale self‑built clusters. Each broker mounts a single SATA/SSD data disk. Scaling is achieved by vertical upgrades—replacing a 5400 RPM drive with a faster 7400 RPM or 10 000 RPM model, or switching to a large‑capacity SSD. However, SSDs are significantly more expensive (≈3× the price of high‑performance cloud HDDs) and have limited write endurance, making this solution unsuitable for long‑term large‑scale growth.

2. Kafka Multi‑Directory Read/Write

Kafka supports multiple log directories via the log.dirs property (e.g., log.dirs=/data,/data1,/data2). Partitions are distributed across these directories, effectively utilizing multiple disks in parallel. This improves I/O capacity without additional hardware, but can suffer from data‑hotspot issues when certain partitions dominate I/O, negating the benefit of multiple disks.

3. RAID Disk Arrays

RAID combines multiple physical disks into a single logical volume. RAID10 (striped mirrors) offers both redundancy and increased throughput (theoretical 2× speed of a single disk). RAID0 provides higher raw I/O but lacks fault tolerance. In practice, RAID’s performance gain is limited by bus bandwidth and other factors, and RAID0’s lack of redundancy makes it risky for production Kafka workloads.

4. LVM Striping

LVM striping works similarly to RAID1/RAID0, providing parallel I/O and, crucially, dynamic expansion via lvextend. All disks must be expanded by the same amount, otherwise the operation fails. While LVM’s dynamic scaling is less useful on physical machines with fixed disks, it shines in cloud environments where virtual disks can be resized on‑demand.

5. Cloud‑Based LVM Solution

In cloud deployments (e.g., Tencent Cloud CVM), each instance can attach multiple cloud disks, which are already replicated and support online expansion. By striping these disks with LVM, a single logical volume is created under /data. This combines the redundancy of cloud disks with LVM’s parallel I/O and easy scaling. For example, a broker can start with six 100 GB disks (≈600 GB total) and later expand each to 200 GB, extending the logical volume to 1.2 TB without service interruption.

Comparison and Recommendations

There is no one‑size‑fits‑all solution:

Small, simple clusters: single‑disk or multi‑directory setups are sufficient.

Large, on‑premise clusters: RAID10 provides a balance of performance and fault tolerance.

Cloud‑native clusters: LVM striping over replicated cloud disks offers the best flexibility and scalability.

Choosing the appropriate disk strategy requires weighing business load, cost constraints, data reliability needs, and the operating environment.

Conclusion

Operating an efficient Apache Kafka cluster hinges on selecting a disk architecture that aligns with workload characteristics and infrastructure constraints. While single‑disk and multi‑directory approaches handle modest demands, RAID10 and cloud‑based LVM solutions are preferable for high‑throughput, large‑scale, or dynamically growing environments.