Designing Distributed File Systems: Solving Local FS Limits

Local file systems such as ext3 and reiserfs manage disk resources and provide a file access interface, but they cannot meet the massive scale of modern internet services, where small files (e.g., product images) or very large files (e.g., video streams) are abundant.

Typical Architecture

A common distributed file system architecture consists of a master (metadata) server, multiple data servers, and many clients. The master may have a standby for failover.

Problems and Solutions

Master Server

Namespace Maintenance – The master maintains the global namespace (directory tree, flat, or graph) and stores auxiliary metadata (file‑to‑block mappings, relationships). Metadata may be kept entirely in memory (e.g., GFS, TFS), in a database (DBFS), or in local files (MooseFS).

Metadata Storage – One simple approach stores the namespace on the master using a local file system such as ReiserFS for small‑file optimization; large files are stored on data servers as fixed‑size blocks (e.g., 64 MiB).

Data Server Management – The master tracks data‑server status via heartbeats (centralized or decentralized). Centralized management (a single master) is widely used (GFS, TFS, MooseFS).

Service Scheduling – Request handling models include single‑thread, one‑thread‑per‑request, and thread‑pool designs; thread pools are most common.

Master‑Backup (HA) – To avoid a single point of failure, a standby master is deployed using HA, UCARP, or virtual IP techniques, with synchronous or asynchronous state replication.

Data Server

Local Storage – Data servers persist file data. Simple mapping stores each file on a single server; most systems use fixed‑size blocks (e.g., 64 MiB) as in GFS, TFS, HDFS. Small files can be packed into larger blocks (e.g., Facebook’s HayStack) or stored via KV stores (Tokyo Cabinet, Redis).

State Maintenance – Servers periodically send heartbeat packets containing CPU, memory, disk I/O, network I/O, and load information to the master for load‑balancing decisions.

Replica Management – Files are replicated across data servers for reliability. Three common replication strategies are: (1) client writes directly to multiple servers, (2) client writes to a primary server that forwards to others, and (3) pipeline replication where each server forwards to the next (used by HDFS, GFS).

Client

Interface – Clients access the file system via POSIX interfaces (implemented through VFS in the kernel or user‑space FUSE). Language SDKs (C/C++, Java, PHP, Python) or RESTful HTTP APIs are also provided.

Cache – To reduce master load and latency, clients cache metadata locally or in external caches (e.g., Tair). Cache consistency can be maintained by client‑side expiration or server‑initiated invalidation; common replacement policies include LRU.

Other Features – Optional encryption, compression, and access‑statistics collection may be added to enhance security and observability.

Summary

The article examines a typical distributed file system architecture, explains core principles such as namespace management, replication, load balancing, and caching, and outlines engineering solutions for the challenges encountered when implementing such systems at scale.