How FastDFS Powers Scalable File Storage for Web Services

FastDFS Overview

FastDFS is an open‑source lightweight distributed file system designed for high‑volume file storage and load‑balancing in services such as photo albums, video platforms, and KYC systems. It stores each file as a single entity on a storage node (no chunking) and provides a proprietary TCP/IP API for upload and download operations.

Key Features

One‑to‑one mapping between logical files and physical files on disk.

Content‑based hash deduplication to avoid storing duplicate files.

HTTP download support via an embedded web server or an external server (e.g., Nginx).

Online linear expansion – new storage volumes (groups) can be added without service interruption.

Built‑in file redundancy and load‑balancing across storage servers.

Metadata (meta‑data) storage on storage nodes; network I/O handled by libevent for high concurrency.

Limitations

No built‑in encryption or access control; files are stored in clear text.

Data synchronization lacks integrity verification, which can lead to silent I/O errors.

Synchronization is single‑threaded and suitable only for files roughly between 4 KB and 500 MB.

Backup count is tied to the number of storage groups; fine‑grained backup configuration is not available.

A failure of a mount point can take down the entire storage node.

No native multi‑data‑center disaster recovery.

Load‑balancing algorithm is static.

System Roles

FastDFS consists of three core components:

Tracker Server : Acts as a scheduler and load balancer. It maintains the list of storage servers, groups, and their health status. Storage servers register with the tracker on startup and send periodic heartbeats. Clients query the tracker to obtain storage node information.

Storage Server : Provides actual file storage and optional backup within a group (also called a volume). Multiple storage servers can belong to the same group, forming a mutual backup set. Files and their meta‑data are persisted on these nodes.

Client : Initiates upload and download requests using FastDFS’s proprietary protocol over TCP/IP. In HTTP scenarios, an Nginx (or similar) reverse proxy is often placed in front of the client.

Upload Workflow

When a storage server starts, it registers with the tracker and periodically reports its capacity and health.

The client contacts any tracker in the cluster to request an upload target. The tracker selects a group (volume) and a specific storage server within that group based on load‑balancing rules.

The client opens a TCP connection to the chosen storage server and sends a write request together with the file data.

The storage server creates a directory for the file, generates a unique file_id, and assembles the final filename.

The storage server returns the file_id, storage path, and generated filename to the client.

The generated filename encodes the storage group, timestamp, and other metadata. Example format (illustrated in the original diagram):

Download Workflow

The client selects any tracker and sends a request containing the target filename.

The tracker parses the filename to extract the group, size, creation time, etc., then chooses an appropriate storage server that holds the file.

The tracker returns the storage server’s address to the client.

The client connects to that storage server and retrieves the file data.

For HTTP‑based downloads, the client can be an Nginx reverse proxy that forwards the request to the selected storage server.

Summary

The article outlines FastDFS’s core architecture, its three roles (tracker, storage, client), and the detailed upload and download processes that bind these roles together. Understanding this high‑level flow enables developers to implement FastDFS‑based file services and to extend the system (e.g., containerized deployment with Docker) as needed.