Write Ahead Log (WAL) Mechanism and Its Application in Distributed Storage Systems

In the operation of storage systems, every data update (creation, deletion, modification) triggers metadata changes that must stay consistent with the physical data; persisting each change directly to an external metadata database can cause high‑frequency I/O, so delayed‑write techniques like Write Ahead Log (WAL) are introduced for optimization.

WAL Overview : WAL records metadata change operations in a log before they are applied to a stable database, thereby reducing the number of direct DB writes and improving efficiency, especially under heavy transaction loads.

WAL also plays a crucial role in disaster recovery: after the metadata DB is loaded, any pending WAL entries can be applied to restore the system to its latest consistent state.

Execution Mechanism : WAL does not store full metadata but change records (e.g., delete, add). These records are first written to a WAL buffer; when the buffer is full or a flush is triggered, the buffered transactions are written to the WAL log. Each operation updates both memory and the WAL buffer. After a WAL segment is applied to the metadata DB, a new commitId marks the latest transaction, allowing older WAL entries to be purged during a checkpoint process (DB + WAL = new DB).

HDFS WAL Application : In HDFS, the WAL concept is realized as the EditLog, while the stable DB corresponds to the standby NameNode's fsimage. HDFS uses a double‑buffering mode to increase transaction throughput. The data flow is Active NameNode → EditLog → Standby NameNode. The standby reads the EditLog in near‑real‑time, periodically checkpoints a new fsimage, and syncs it back to the active node; old EditLog files are purged based on commit transaction IDs.

WAL Apply Control : During disaster recovery, applying WAL entries may encounter corrupted records. Users can choose to abort the apply process or ignore specific errors to achieve the best possible data restoration.

In summary, WAL provides an efficient, reliable way to persist metadata changes, improve throughput, and support recovery in large‑scale distributed storage systems.