Why Distributed File Systems Matter: Understanding CAP and MogileFS

Distributed File System Overview

What is Distributed Storage?

Distributed storage systems spread data across multiple independent devices, avoiding the bottleneck of centralized servers and improving reliability, availability, and scalability for large‑scale applications.

Design Goals of Distributed File Systems

Access transparency

Location transparency

Concurrency transparency

Failure transparency

Hardware transparency

Scalability

Replication transparency

Migration transparency

CAP Theory

C : Consistency – every read returns the most recent write.

A : Availability – every operation returns a response within a bounded time.

P : Partition tolerance – the system continues to operate despite network partitions.

Eric Brewer proved that at most two of these properties can be simultaneously guaranteed, leading to the well‑known CAP theorem.

Relational databases typically favor CA, while many NoSQL key‑value stores favor AP.

Consistency Models

Strong consistency (ACID) – hard to achieve in distributed environments due to performance costs.

Weak consistency, including eventual consistency – the system may return stale data within an inconsistency window.

Final consistency – a special case of weak consistency where updates eventually converge to the latest value.

Server consistency parameters: N (number of nodes), W (writes required), R (reads required). Typical rules: W+R>N for strong consistency, W=N,R=1 for read‑optimal, W=1,R=N for write‑optimal, W+R≤N for weak consistency.

Types of Distributed Storage

Centralized: NAS, SAN

Distributed: dedicated metadata nodes (centralized metadata, data nodes store data) or no dedicated metadata nodes (metadata stored with data).

Common Distributed File Systems

GFS (Google File System) – optimized for large files.

HDFS (Hadoop Distributed File System) – derived from GFS, also suited for large files.

TFS (Taobao File System) – open‑source, handles massive small files, stores metadata in a relational DB.

GlusterFS – decentralized design, good for large files.

Ceph – Linux kernel‑integrated, PB‑scale distributed file system.

MogileFS – open‑source system for building distributed file clusters.

MogileFS Introduction

Overview

MogileFS is an open‑source distributed file system created by Danga Interactive (the same team behind Memcached and Perlbal). It is widely used by many large websites for storing massive amounts of image data.

Key Features

Application‑level service without core component dependencies.

No single point of failure; consists of trackers, storage nodes (mogstored), and a database.

Automatic file replication at the class level.

Transport‑neutral; works over NFS or HTTP.

Flat namespace using domains instead of directories.

No shared data required.

Components

Tracker – the scheduler (mogilefsd) that manages metadata in a database, handles deletions, replication, monitoring, and queries.

Database – stores all metadata; high availability is essential.

Mogstored – storage node, typically a WebDAV server (default on port 7500) that handles file creation, deletion, and retrieval.

Utilities such as mogadm and mogtool interact with trackers, and client APIs exist for Perl, PHP, Java, Python, etc.

Basic Workflow

1. The client requests to open a file, notifying a tracker to obtain an available storage location.

2. The tracker performs load balancing and returns possible locations.

3. The client writes to one of the locations; on failure it retries elsewhere.

4. The client sends a “create_close” request to the tracker, indicating where the file was stored.

5. The tracker records the filename‑domain mapping in the database.

6. The tracker initiates background replication according to the file’s class policy.

7. The client requests “get_paths” for the file; the tracker returns a list of URLs based on node load and health.

8. The client attempts the URLs in order; the tracker continuously monitors node health to avoid dead connections.

Full MogileFS Architecture

1) Servers : mogilefsd (tracker) and mogstored (storage node). The tracker stores metadata in the database, while mogstored provides a WebDAV interface for file operations.

2) Utilities : management tools such as mogadm.

3) Client APIs : libraries for various languages enabling applications to store and retrieve files.