Why Small Files Are a Problem in Big Data and How Delta Lake Compaction Solves It

The article discusses the long‑standing issue of numerous small files in traditional data warehouses and the reasons they arise, such as fine‑grained partitioning, continuous incremental writes, and multiple storage states during updates.

It explains why a large number of files becomes a bottleneck, primarily because HDFS stores file metadata in the memory of a single node, leading to scalability limits despite later federation attempts.

The author describes how Delta Lake can address the small‑file problem through compaction, noting that while Delta currently lacks built‑in compaction, its extensible API allows custom implementations that avoid affecting ongoing reads and writes.

A specific Delta‑based compaction implementation is outlined, with the constraint that tables containing update/delete (upsert) operations cannot be compacted; only append‑only tables are safe because their existing files never change.

The compaction workflow consists of reading data up to a target version, physically deleting files marked for removal, merging added files per partition into larger files, and finally acquiring a transaction and committing the changes.

The author, William Zhu, a senior data architect with 11 years of R&D experience, concludes with a friendly request for likes, shares, and follows.