Design Considerations and Architecture of Distributed File Systems

Distributed file systems are a foundational technology in the storage domain, with HDFS and GFS being the most well‑known examples. Understanding their design principles is valuable for solving similar problems in future scenarios.

Historically, distributed file systems date back to the 1980s, exemplified by Sun's Network File System (NFS), which abstracted remote disks as network resources, enabling larger capacity, host failover, data sharing, backup, and disaster recovery.

With the rise of the Internet, the focus shifted to massive storage capacity, fault tolerance, high availability, persistence, and scalability, because commodity servers are less reliable than dedicated storage machines.

Key requirements for a distributed file system include:

POSIX‑compatible file interface for ease of use and legacy compatibility.

Transparency to users, behaving like a local file system.

Data persistence to prevent loss.

Scalability to accommodate growing data pressure.

Robust security mechanisms to protect data.

Consistency so that reads return the same content regardless of timing.

Additional desirable features are large capacity, high concurrency, high performance, and efficient hardware utilization.

Typical architectural components:

Storage component: stores file data, ensures persistence, replica consistency, and block allocation/merging.

Management component: maintains metadata (file locations, sizes, permissions) and monitors storage node health and data migration.

Interface component: provides SDKs (Java/C/C++), CLI, and FUSE mounting for applications.

Two main deployment models exist: centralized (with a master node) and decentralized (master‑less).

1. Centralized architecture (e.g., Google File System): A master node stores metadata and coordinates chunkservers. Clients query the master for chunk locations, then communicate directly with the appropriate chunkservers for data transfer. The master reduces its load by returning only metadata, not participating in data reads/writes.

2. Decentralized architecture (e.g., Ceph): All nodes are autonomous; the cluster consists of a single node type that stores both metadata and data (RADOS). Ceph uses the CRUSH algorithm to map files to storage nodes without a central coordinator.

Persistence and replication: Data is typically persisted via multiple replicas. Consistency can be achieved through synchronous writes (all replicas must acknowledge) or optimized approaches such as parallel writes, chain writes, or quorum‑based writes (W+R>N).

Replica placement must consider fault domains to avoid losing all copies in a single failure, often by spreading replicas across different racks or data centers. Detection of corrupted or stale replicas relies on checksum/version checks performed by storage nodes reporting to a monitor (centralized) or a small monitor cluster (Ceph).

When selecting a replica for a client read, strategies include round‑robin, fastest node, highest success rate, lowest CPU load, or proximity.

Scalability: Adding storage nodes involves registration with the master (or monitor) and rebalancing data. Load balancing can be based on disk usage, CPU, and network metrics. New nodes may be gradually warmed up to avoid overload. Master scalability can be improved by using larger block sizes, multi‑level masters, or stateless masters sharing a common metadata store (e.g., iRODS).

High availability: Master HA requires replication of metadata, often via a standby node or shared storage (RAID1). Storage node HA is achieved through replica redundancy. Persistent metadata can be stored in databases or log‑based files with periodic snapshots.

Performance optimization and cache consistency: Techniques include in‑memory caching, prefetching, and request batching. Caching introduces consistency challenges such as lost updates and stale reads, which can be mitigated by read‑only files, locking mechanisms, or configurable lock granularity.

Security: Access control models include DAC (Unix‑style user/group/privilege), MAC (mandatory labels), and RBAC (role‑based). Distributed file systems may adopt or extend these models; for example, Ceph adds its own DAC‑like permissions, while Hadoop can integrate Apache Sentry for RBAC.

Additional considerations:

Space allocation: contiguous vs. linked‑list allocation, with indexing (i‑nodes) to mitigate fragmentation.

File deletion: immediate vs. delayed (logical) deletion, with background reclamation.

Small‑file handling: store small files as metadata pointing to large data blocks, preserving block efficiency.

Deduplication: use file fingerprints (MD5, SHA‑256, SimHash, MinHash) to identify identical content for storage savings and integrity checks.

In summary, designing a distributed file system involves balancing persistence, scalability, availability, performance, and security. The article outlines the major problems and possible solutions, providing a foundation for further detailed research and implementation.