Design and Considerations of Distributed File Systems

Distributed file systems are a fundamental component of modern distributed computing, with HDFS and GFS being the most well‑known examples.

The article begins with a brief history, noting early systems like Sun's NFS that separated disk storage from hosts, and explains how the rise of the internet shifted focus to massive storage capacity, fault tolerance, high availability, persistence, and scalability.

Key requirements for a distributed file system are listed, including POSIX‑compatible interfaces, transparency to users, data persistence, scalability, reliable security mechanisms, and consistent read results, as well as desirable optional features such as large capacity, high concurrency, fast performance, and efficient hardware utilization.

The architecture is examined in two main models: centralized systems (e.g., GFS) where a master node manages metadata, location, and fault detection, and decentralized systems (e.g., Ceph) where all nodes are autonomous and the CRUSH algorithm determines data placement.

Persistence is achieved through multi‑replica strategies, with discussions on synchronous writes, parallel or chain writes, and quorum‑based approaches (W+R>N) to balance consistency and performance.

Scalability considerations include load‑balanced storage node addition, pre‑warming new nodes to avoid overload, and transparent data migration handled either by a central controller or by logical‑physical separation in decentralized designs.

High availability is addressed for both master nodes (using replication, shared storage, or multi‑level metadata hierarchies) and storage nodes (ensuring replica durability and rapid recovery).

Performance optimizations cover in‑memory caching, prefetching, request batching, and the trade‑offs between caching benefits and consistency challenges, with solutions such as read‑only files, locking mechanisms, and appropriate lock granularity.

Security is discussed through various access control models—DAC, MAC, and RBAC—and how distributed file systems integrate or extend these models (e.g., Ceph's DAC variant, Hadoop's reliance on OS permissions, and Apache Sentry's RBAC).

Additional topics include space allocation strategies (contiguous vs. linked), file deletion policies (immediate vs. delayed logical deletion), handling of small files, and deduplication using file fingerprints (MD5, SHA‑256, SimHash, MinHash).

The article concludes that distributed file system design is complex and context‑dependent, urging readers to consider the presented factors when evaluating or building such systems.