How Switching S3 Compression to Zstd Cut Storage Costs by 30%

Former AWS Vice President Adrian Cockcroft revealed on Twitter that Amazon switched the compression algorithm used for S3 logs from gzip to Zstandard (zstd), saving roughly 30% of storage space, equivalent to exabytes.

Amazon employees clarified that the change affects the way S3 stores its own service data (primarily logs), not customer data, and the switch to zstd reduced storage costs by about 30%.

Although AWS did not officially announce the change, a price reduction of 31% at the end of November 2021 aligns with the reported savings.

About Zstd

Zstandard is an open‑source, fast, lossless compression algorithm developed by Facebook, targeting real‑time compression scenarios with better ratios than zlib. It can trade compression speed for higher ratios, and it offers a special “dictionary compression” mode for small data, where a trained dictionary improves compression efficiency.

Zstd aims to combine high compression performance and ratio, distinguishing it from algorithms such as zlib, lz4, and xz. It also supports dictionary compression for small data blocks.

Below is a typical performance comparison of common compression algorithms:

Reference: https://twitter.com/adrianco/status/1560854827810361345