Message Compression at Scale: From No Compression to Selective Strategies

One Late‑Night Bandwidth Alert

At 2 am an alarm shows cross‑datacenter link saturation and producer latency jumping from 5 ms to 300 ms. Grafana reveals inbound traffic rising from 2 Gbps to 8 Gbps while QPS only increases 1.2×, caused by a business changing a message format from a 800‑byte JSON to a 4 KB payload. When asked why compression wasn’t enabled, the answer was that enabling it once pushed CPU to 90 % and the change was rolled back.

Never‑Compress Era: The Sweet Spot of Simplicity

New systems typically keep compression disabled because the bandwidth savings are modest while the added complexity is certain. Compressing a 500‑byte JSON to ~200 bytes saves ~60 % size, but each message costs an extra 10 µs to compress and 5 µs to decompress. At 864 billion messages per day this translates to roughly 25 TB of bandwidth saved, worth a few thousand dollars per month—often not enough to justify two weeks of engineering effort.

The Turning Point: When Bandwidth Becomes the Bottleneck

Once traffic exceeds a certain scale, the cost shifts from a few thousand dollars to a full‑link choke. A 10 GbE NIC can sustain about 800 MB/s after protocol overhead; with 2 KB average messages this caps a broker at ~400 k QPS. Achieving 10 M QPS therefore requires ~25 brokers just to absorb network bandwidth, not CPU or disk. Cross‑datacenter links are even more expensive, making compression a necessity rather than an optimization.

Choosing the Right Algorithm

Four common algorithms are compared on compression ratio versus speed:

LZ4 / Snappy : fastest, low‑memory, ~45‑50 % compression ratio; unsuitable for already binary formats.

Gzip : high ratio, slow; best for offline or asynchronous workloads.

Zstd : Facebook’s open‑source algorithm, ratio comparable to gzip with speed close to LZ4; supports adjustable levels. Kafka supports Zstd from version 2.1; level 3‑6 offers ~40 % better ratio than Snappy with only ~20 % slower speed.

Recommendation: for new deployments after 2020, default to Zstd.

Where to Apply Compression

Compression can be performed at three points:

Producer side (default): saves bandwidth on the wire, broker disk, and consumer ingress, but adds CPU load on producers, which may be problematic for weak edge devices.

Broker side : saves disk space but does not reduce the costly producer‑to‑broker bandwidth, and shifts all CPU cost to the broker; rarely used except when producers are extremely constrained.

Business‑layer pre‑compression : the application compresses payloads (e.g., with Zstd) before sending; allows fine‑grained control but requires handling compression failures and version compatibility.

Most teams adopt a producer‑side variant with batch compression.

Batch Compression: Compressing by the RecordBatch

Compressing a single small message often yields a worse ratio (e.g., a 200‑byte JSON may expand to 220 bytes with Zstd) because the compressor needs metadata. Batch compression leverages repeated field names across many messages; a batch of 100 order messages can replace repeated field names with short back‑references, dramatically improving ratio. Kafka’s batch.size and linger.ms control batch formation. Larger batches achieve higher compression, but low‑latency producers that set linger.ms=0 may end up with tiny batches and poor ratios.

Selective Compression: Not All Messages Need It

Three scenarios where compression offers little or negative benefit:

Already binary‑encoded messages (e.g., Protobuf) – typical gain 1.2‑1.4×, but CPU cost remains.

Ultra‑low‑latency messages (trading, real‑time gaming) – extra tens of microseconds exceed latency budgets.

High‑entropy payloads (base64‑encoded media, encrypted data) – compression cannot reduce size and may increase it.

Thus, compression policies should be defined per‑topic.

Three‑Layer Implementation of Selective Compression

Layer 1 – Cluster default : set a default algorithm (e.g., Zstd level 3 or Snappy) so new services work out‑of‑the‑box and capture ~70 % of potential savings.

Layer 2 – Topic‑level overrides : for topics that need special handling (e.g., trading), explicitly set compression.type=none or compression.type=lz4 in metadata, allowing platform‑wide configuration without code changes.

Layer 3 – Producer‑side dynamic decisions : at runtime skip compression for messages < 500 bytes, for high‑entropy payloads, or when producer CPU exceeds a threshold.

This layered approach avoids a chaotic mix of configurations.

Compression Dictionaries: An Overlooked Multiplier

Zstd can load an external dictionary trained on historical data, containing the most frequent byte sequences of a topic. In small‑batch scenarios this can add 30‑50 % extra compression; in large batches, 10‑20 % extra. Managing dictionaries requires periodic retraining, version alignment between producers and consumers, and infrastructure for distribution and hot‑updates, so it is typically adopted only by high‑traffic services.

Observability: Compression Is a Hidden Component

Without dedicated metrics, compression problems go unnoticed. Real‑world issues include:

Message format change dropping compression ratio from 3× to 1.2×, leading to unexpected 30 TB extra disk usage.

QPS surge causing producer CPU saturation by compression, resulting in timeouts and OOM.

Broker upgrade switching default algorithm from Snappy to Zstd, breaking older consumers.

Metrics that expose compression ratio, CPU usage, and latency turn compression from a black‑box into an observable engineering component.

Evolution Roadmap: From No Compression to Platform‑Level Capability

Four stages illustrate the progression:

100 k QPS : No compression; focus on rapid feature delivery.

1 M QPS : Enable a cluster‑wide default (Snappy/LZ4) to reduce problem domain.

10 M QPS : Introduce selective compression with topic overrides and producer‑side dynamics.

Multi‑business platform : Productize compression – pre‑trained dictionaries, hot algorithm switching, visual monitoring.

Each stage is driven by rising cost and stability pressures rather than pure technical curiosity.

Engineer’s Three‑Step Remedy

Configure the offending topic with Zstd level 6, boosting compression from 1.0× to 3.8× and instantly lowering bandwidth.

Add a dynamic downgrade in the producer SDK: if CPU > 70 % drop to level 3, then to LZ4, finally to no compression.

Expose compression ratio and CPU usage on the core monitoring dashboard and generate weekly reports for topic owners.

These actions saved ~20 % cross‑datacenter bandwidth and extended broker disk capacity by six months.

Conclusion

Message compression is not a binary “on/off” decision but a series of trade‑offs involving algorithm, placement, batch size, and selectivity. The true architectural skill lies in choosing the right combination for the given traffic scale, cost constraints, and stability requirements.