Exploring Today’s Popular Distributed File Systems: Lustre, Hadoop, FastDFS, and More

This article gives a brief introduction to a number of currently popular distributed file systems.

Lustre (www.lustre.org)

Lustre is a large‑scale, highly available cluster file system originally developed by Sun Microsystems. It is open‑source (GPL) and can support over 10,000 nodes and petabytes of storage. It runs on Linux and is written in C/C++.

Hadoop (hadoop.apache.com)

Hadoop is not only a storage system but a framework for running distributed applications on large clusters of commodity hardware. It is licensed under Apache, written in Java, and can be resource‑intensive.

MogileFS (www.danga.com)

MogileFS is an open‑source distributed file system with features such as application‑level components, no single point of failure, automatic file replication, and reliability better than RAID. It runs on Linux and is written in C/C++.

FreeNAS (www.openqrm.org)

FreeNAS is a NAS‑oriented operating system based on FreeBSD, providing disk management, RAID, and support for protocols like FTP, NFS, RSYNC, CIFS, AFP, UNISON, and SSH. It enables repurposing old hardware as a NAS server.

FastDFS (code.google.com/p/fastdfs)

FastDFS is an open‑source distributed file system that manages file storage, synchronization, and access (upload/download). It uses a tracker for load balancing and storage nodes for file handling, and also manages file metadata as key‑value pairs. It is written in C/C++ and runs on Linux.

NFS (www.tldp.org/HOWTO/NFS-HOWTO/index.html)

The Network File System (NFS) allows remote directories and files to be accessed as if they were local, reducing local disk usage and enabling shared home directories and peripheral devices across the network. It is written in C/C++ and is cross‑platform.

OpenAFS (www.openafs.org)

OpenAFS is an open‑source distributed file system that organizes servers into cells, providing transparent file sharing across LAN and WAN. It uses large client caches, Kerberos‑based security, and ACLs for fine‑grained access control. It follows the IBM Public license and runs on Linux.

MooseFS (derf.homelinux.org)

MooseFS is a fault‑tolerant networked distributed file system that distributes data across multiple servers and presents a unified Unix‑like namespace via FUSE. It is written in Perl and is cross‑platform, though it does not fully eliminate single points of failure.

pNFS (www.pnfs.com)

Parallel NFS (pNFS) extends the traditional NFS protocol to provide parallel file access, dramatically increasing transfer rates for high‑performance computing workloads. It is written in C/C++ and runs on Linux.

GoogleFS

GoogleFS is a scalable, fault‑tolerant distributed file system designed for large‑scale data‑intensive applications. It runs on inexpensive commodity hardware and offers high performance to many users.