Design Principles and Architecture of Distributed File Systems

Distributed file systems are a foundational application in the distributed domain, with HDFS and GFS being the most well‑known examples. Understanding their design principles helps when tackling similar scenarios.

1. Overview

Early distributed file systems like Sun's 1984 NFS separated disk storage from hosts, enabling larger capacity, host switching, data sharing, backup, and disaster recovery. As the Internet era brought massive data growth, requirements shifted to massive storage, fault tolerance, high availability, persistence, and scalability.

2. Historical Systems

NFS used TCP/IP to forward file commands from clients to remote servers, making the process transparent to users. Over time, the need for distributed storage and reliable commodity servers introduced new challenges.

3. Requirements

POSIX‑compatible file interface

Transparency to users

Persistence (no data loss)

Scalability (elastic capacity)

Robust security mechanisms

Data consistency

Additional desirable features include large storage space, high concurrency, fast performance, and efficient hardware utilization.

4. Architectural Models

Components needed:

Storage component: stores file data, ensures persistence, replication, and block management.

Management component: maintains metadata (file locations, sizes, permissions) and monitors storage node health.

Interface component: provides SDKs, CLI, or FUSE mounting.

Two main deployment routes exist:

4.1 Centralized Architecture (e.g., GFS)

The master node handles metadata, fault detection, and data migration. Clients query the master for file locations, then directly contact the appropriate chunkservers for data transfer.

Advantages include simplified control and strong functionality, but the master can become a bottleneck.

4.2 Decentralized Architecture (e.g., Ceph)

All nodes are autonomous; the cluster consists of a single node type. Ceph uses the CRUSH algorithm to map objects to storage nodes without a central manager.

5. Persistence

Data durability is achieved via multiple replicas. Key challenges include ensuring replica consistency, dispersing replicas to avoid correlated failures, detecting corrupted or stale replicas, and selecting the appropriate replica for client reads. Techniques such as synchronous writes, parallel writes, chain writes, and quorum‑based writes (W+R>N) are discussed.

6. Scalability

When adding storage nodes, they register with the master, which can then allocate new blocks. Considerations include load balancing, preventing overload on newly added nodes, and transparent data migration. Centralized systems handle migration via the master, while decentralized systems rely on logical‑physical separation (e.g., Ceph's placement groups).

7. High Availability

Both metadata and storage nodes require HA. Metadata can be replicated to a standby node or stored on shared storage (e.g., RAID1). Storage node HA follows from the persistence mechanisms that keep replicas alive.

8. Performance Optimization & Cache Consistency

Common optimizations: in‑memory caching, prefetching data blocks, and batching read/write requests. Caching introduces consistency challenges such as lost updates and stale reads, mitigated by read‑only files, locking mechanisms, and appropriate granularity of locks.

9. Security

Distributed file systems serve multiple tenants and must enforce security. Common access control models include DAC (Unix‑style), MAC (e.g., SELinux), and RBAC. Systems like Ceph and Hadoop integrate these models, sometimes extending them.

10. Additional Topics

Space allocation can be contiguous or linked‑list based; each has trade‑offs. Index tables (i‑nodes) mitigate linked‑list drawbacks. File deletion may be immediate or delayed; most systems use logical deletion with eventual reclamation. Small‑file workloads benefit from storing metadata that points to large data blocks. File fingerprinting (e.g., MD5, SHA‑256, SimHash) enables deduplication and integrity checks.

11. Conclusion

Distributed file systems involve many complex considerations beyond those covered here. This overview highlights key design questions and possible solutions, serving as a starting point for deeper investigation.