The First Four Chaos Experiments to Run on Apache Kafka

Apache Kafka is an open‑source distributed messaging platform that handles trillions of records per day with high throughput and low latency, providing fault‑tolerance, replication, and automatic disaster recovery.

Because Kafka is a critical data pipeline, reliability is essential. Potential failure modes include broker interruptions, ZooKeeper failures, and upstream/downstream application faults.

Chaos engineering allows proactive testing of these failure modes before they occur in production. Using the Gremlin SaaS platform, four chaos experiments are designed and run on a Confluent Kafka cluster.

Experiment 1 – Broker Load Impact on Processing Latency : Disk I/O load is increased on brokers while a Kafka Music demo app streams data. The hypothesis is that higher I/O will reduce throughput; results show throughput remains stable even above 150 MB/s, but monitoring of I/O utilization is recommended.

Experiment 2 – Message Loss Risk : A black‑hole attack blocks network traffic to the broker leader, simulating leader failure. The hypothesis is that some messages will be lost but Kafka will elect a new leader and continue replication. Results confirm no data loss when acks=all is used, though latency may increase.

Experiment 3 – Split‑Brain Prevention : Multiple brokers are shut down simultaneously using Gremlin’s shutdown attack to test cluster behavior under majority failure. The hypothesis is that the cluster will temporarily stop throughput but brokers will re‑join without split‑brain. Results show reduced performance and temporary leader election, but the cluster recovers without data inconsistency.

Experiment 4 – ZooKeeper Interruption : A black‑hole attack drops all traffic to ZooKeeper nodes, testing Kafka’s ability to survive a ZooKeeper outage. The hypothesis is that Kafka can continue processing messages while ZooKeeper is down, though cluster state changes are paused. Results confirm message flow continues, but cluster management is delayed until ZooKeeper recovers.

Overall, the experiments demonstrate that systematic chaos testing can reveal reliability gaps, guide mitigation strategies, and increase confidence in Kafka deployments.