Rack Awareness: From Zero to High‑QPS – Boost Availability, Cut Cross‑AZ Traffic

One Rack‑Power Outage Wiped All Three Replicas

On a Saturday afternoon a 30‑broker Kafka cluster used for core order processing suffered a power‑cable trip that cut power to five adjacent racks (A02‑A06). Six brokers were located in those racks, so 30% of the topic partitions reported NotEnoughReplicasException and the system was unavailable for 47 minutes. Investigation revealed that the three replicas of each affected partition had been randomly placed in the same rack group, proving that three replicas do not guarantee fault‑domain availability.

What Rack Awareness Actually “Senses”

Rack awareness injects failure‑domain information into the system. A failure domain is a set of resources that fail together (rack power loss, switch failure, AZ network partition). Without explicit tags the scheduler assumes brokers are independent, leading to unsafe replica placement.

The core idea is to translate invisible physical topology into logical labels consumable by the scheduler. Labels can be coarse (Region/AZ) or fine‑grained (rack, switch, power circuit). The granularity directly limits the replica‑placement algorithm.

How Replicas “Cluster” Without Rack Awareness

Traditional replica‑assignment aims to balance leader and partition counts across brokers. Early Kafka algorithms simply chose a start broker and round‑robin‑assigned N brokers for each partition. This works only when broker IDs have no physical correlation. Three common scenarios cause clustering:

New servers bought in bulk and installed in adjacent racks produce consecutive broker IDs, so round‑robin placement keeps replicas together.

During scale‑out, the reassign tool prefers the least‑loaded brokers, which are often the newly added ones in the same rack.

When brokers are spread across AZs but rack awareness is disabled, the algorithm may still place two replicas in AZ‑A and one in AZ‑B, leading to ineffective AZ‑level isolation.

Probability of all three replicas landing in the same AZ with 3 AZs and 10 brokers per AZ is 3 × C(10,3) / C(30,3) = 8.86%, i.e., roughly one out of twelve partitions becomes unavailable during an AZ failure.

Tag‑Injection Modes

Four practical ways to attach rack tags to brokers:

Static config – set broker.rack (Kafka) or broker startup config (Pulsar). Stable but error‑prone during scale‑out if operators forget to update.

Startup script – fetch topology from CMDB via environment variables and inject at launch. Adds a dependency on CMDB availability.

Runtime query – brokers periodically request their label from a topology service (common in Pulsar, HBase). Requires HA for the service.

IP‑segment inference – network design assigns a unique /24 per rack; routers infer rack from IP. Works for large internet firms but needs strict network planning.

Most teams combine static config with a deployment‑time sanity check that validates the CMDB‑provided label before release, keeping the source of truth in CMDB while avoiding stale config.

Replica‑Placement Algorithm: From Round‑Robin to Topology‑Aware

Kafka introduced rack‑aware replica assignment in version 0.10. The algorithm proceeds in three steps:

Interleave brokers by rack so adjacent positions belong to different racks.

For each partition, pick N consecutive positions from the interleaved list; because the list is interleaved, the N brokers span N distinct racks.

Rotate the start index for each partition to avoid load hotspots.

If each rack has a comparable number of brokers, the algorithm guarantees strict rack isolation for N replicas without randomization. Limitations include:

Fewer racks than replicas – unavoidable co‑location.

Highly uneven broker counts per rack – leader balance is possible but follower distribution may skew.

Adding a new rack does not automatically rebalance existing partitions; manual kafka-reassign-partitions is required.

Pulsar adds weight‑based randomization to tolerate heterogeneous rack sizes, while RocketMQ 4.x relied on manual broker‑name placement and 5.x introduced a controller‑driven topology awareness.

Leader Distribution: The Second Battle

Even if followers are rack‑isolated, placing all leaders in the same rack causes N simultaneous leader failovers during a rack outage, each incurring seconds of write latency. Kafka’s preferred‑leader election periodically moves leaders to their “preferred” broker, and rack awareness influences the preferred‑leader selection to spread leaders across racks.

However, uniform leader count does not equal uniform traffic. Hot topics concentrated in one rack can saturate network and CPU. Mitigations include:

Basic leader count balancing (supported by most MQs).

Traffic‑weighted leader balancing (requires monitoring feedback).

Manual reassign of hot‑topic leaders as a fallback.

Pulsar goes further by weighting read and write traffic separately and considering inter‑rack bandwidth in its load‑balancer, emphasizing that rack awareness also serves traffic‑scheduling.

Consumer Proximity: Rack‑Aware Reads

Traditional consumers always read from the partition leader, incurring cross‑AZ traffic when the leader resides in a different AZ. Kafka 2.4 (KIP‑392) introduced client.rack, allowing a consumer to declare its rack. The broker then prefers a follower in the same rack; if none exists, it falls back to the leader.

Trade‑offs:

Consistency – followers may lag by hundreds of ms; Kafka exposes HighWatermark to ensure only fully‑ISR data is read, at the cost of higher latency.

Bandwidth savings – replication traffic stays the same, but consumer‑side traffic can drop 50‑70% when consumer count far exceeds replica count.

Failure fallback – if the local follower becomes unavailable, the consumer switches back to the leader, causing a brief latency spike.

In practice many deployments also reduce acks from all to 1 to lower latency, accepting weaker durability.

Hidden Cost: Cross‑Rack/ AZ Replication Bandwidth

Strict rack isolation increases inter‑rack replication traffic. Writing 100 MB/s with three replicas confined to three AZs generates 200 MB/s of cross‑AZ traffic (each message replicated twice across AZs). At GB/s write rates, cross‑AZ traffic can reach several GB/s, leading to costly cloud‑provider charges.

Mitigation strategies:

Reduce replica count (e.g., from three to two) – cuts bandwidth but lowers fault tolerance.

Enable compression (GZIP/LZ4/ZSTD) – reduces payload size by 60‑80% at CPU cost.

Adjust replica granularity – e.g., two replicas across AZs plus one intra‑AZ replica, sacrificing perfect isolation for bandwidth savings.

Higher replication also lengthens acknowledgment latency (0.1 ms intra‑rack RTT vs. 1‑3 ms cross‑AZ), prompting some latency‑sensitive workloads to switch from acks=all to acks=1.

Typical Rack‑Awareness Failure Scenarios

Scale‑out without rebalancing – new AZs or racks are added but existing topics are not reassigned, leaving old partitions unprotected.

Missing or wrong rack label – a broker without broker.rack is treated as “unknown” and falls back to random placement, silently breaking rack isolation.

Inaccurate topology data – CMDB says a broker is in AZ‑A while it physically resides in AZ‑B; periodic traceroute checks are needed.

Incorrect failure‑domain boundaries – treating an entire data center as one AZ while shared power circuits create sub‑AZ failure domains.

These issues highlight that rack awareness is a “contract‑based reliability” mechanism; its effectiveness hinges on the correctness of the tags.

Evolution Path: From Single‑Rack to Multi‑Region

Three scaling stages are described:

Start‑up (sub‑million QPS) – single data center, rack‑level isolation; tags like rack-01 ‑ rack-30.

Million‑QPS stage – 2‑3 AZs, rack‑plus‑AZ tags; first noticeable cross‑AZ bandwidth cost; introduce follower reads and compression.

Ten‑million QPS stage – region‑level disaster recovery; tags expand to Region‑A, AZ‑B, rack‑C; cross‑region replication uses asynchronous mirroring (MirrorMaker, Pulsar Geo‑replication) because latency is too high for synchronous replication.

Each jump requires revisiting failure‑domain definitions and often splitting a single cluster into multiple clusters with coordinated replication.

Supporting Capabilities

Chaos testing – periodically shut down a rack to verify that the system remains available.

Replica health visualization – heat‑map dashboards showing per‑partition rack distribution.

Impact analysis before changes – tools like bin/pulsar-admin topics ensemble-placement-policy or Kafka’s Cruise‑Control predict post‑change replica layouts.

Cross‑cluster coordination – when an AZ fails, a backup region must detect the event and take over traffic.

From “Configure‑and‑Forget” to Continuous Guardrails

After the initial outage the company instituted three practices: (1) daily health checks that verify each partition’s replicas span distinct racks, (2) an approval workflow for any rack or broker relocation that includes automated topology validation, and (3) quarterly simulated rack‑power failures.

Three years later, repeated power‑loss drills show stable availability, confirming that the technical hurdle of rack awareness is modest; the real challenge is embedding it into an ongoing reliability culture.

Ultimately, rack awareness evolves from a simple configuration to a multi‑layered topology orchestration that enables fault‑domain isolation at rack, AZ, and region levels, turning a high‑availability feature into a cost‑optimization and resilience strategy.