From Live CD to Docker: The Evolution of Union Filesystems and Modern Image Formats

Docker originated in 2013 from dotCloud, initially built on LXC containers and the aufs storage driver, a concept that traces back to the Live CD era where operating systems were packed onto a CD or USB without installation.

In the early 2000s, Knoppix introduced the revolutionary Live CD concept, compressing a full Linux system into a 700 MB disc using a compressed loop device (cloop) that allowed on‑the‑fly decompression, later evolving to use SquashFS for more efficient block‑level compression.

The key innovation was the introduction of a read‑only layer combined with a writable overlay, implemented via union filesystems such as UnionFS, aufs, and later overlayfs, enabling a complete OS image to be treated as an immutable base with a mutable top layer.

Knoppix’s cloop device used an index to map compressed blocks, but suffered from slower random reads and CD‑ROM seek times; SquashFS improved performance by indexing and decompressing only needed blocks. Example paths illustrate the union mount behavior: /usr/local/ resides on one layer while /usr may be on another, and modifications to /etc/passwd appear in the top writable layer.

Docker revived union filesystems under the slogan “Make UnionFS Great Again,” treating each container image as a stack of layers where the base layer is read‑only and subsequent layers add changes, providing a clear developer experience through Dockerfiles that map directly to these layers.

Despite Docker’s success, the OCI v1 image format faces challenges: content redundancy across layers, lack of parallel transfer, inability to verify small data blocks, and difficulty handling cross‑layer deletions. Studies show only about 6 % of image data is actually used, motivating a shift away from layer‑centric designs.

The next‑generation image system proposes lazy loading of compressed data blocks, similar to SquashFS, allowing on‑demand fetching of file contents and integrity verification without downloading the full image, and can be integrated with technologies like virtio‑fs, runC, and Kata Containers.

Author: Wang Xu, senior technical expert at Ant Group, core contributor to Kata Containers and Docker‑related open‑source projects, with extensive experience in cloud computing and container standards.