Design Principles and Architecture of the FastDFS Distributed File System

FastDFS is an open‑source lightweight distributed file system composed of a tracker server, storage servers, and client libraries, primarily solving massive data storage for online services that handle medium‑sized files (4 KB < file size < 500 MB).

Storage server organizes data by groups (also called volumes). Each group contains multiple storage machines that replicate each other's data; the group's usable capacity is limited by the smallest storage node, so it is recommended to configure storages in a group with similar capacities to avoid waste.

Grouping simplifies application isolation, load balancing, and replica count customization—different applications can be placed in separate groups for isolation and balanced distribution based on access patterns. However, a group's total capacity is bounded by the capacity of a single machine, and recovery time can be long if a node fails.

Each storage node relies on the local file system and can be configured with multiple data directories (e.g., /data/disk1‑/data/disk10). When a write request arrives, the storage selects one of these directories according to configured rules and creates a two‑level sub‑directory structure (256 × 256) to spread files, routing each new file by hash into a specific sub‑directory.

Tracker server acts as the coordinator, maintaining an in‑memory map of groups to storage server lists. Storages register their group information and send periodic heartbeats; the tracker does not persist any data, making it easy to scale by adding more tracker nodes that operate as a fully equal cluster.

Upload file – FastDFS provides basic file operations (upload, download, append, delete) through client libraries. The upload process involves several selection steps:

当集群中不止一个 tracker server 时，由于 tracker 之间是完全对等的关系，客户端在 upload 文件时可以任意选择一个 tracker。

Choosing a storage group follows rules such as round‑robin, specified group, or load‑balance (prefer groups with more free space).

当 tracker 接收到 upload file 的请求时，会为该文件分配一个可以存储该文件的 group，支持如下选择 group 的规则：1. Round robin，2. Specified group，3. Load balance，剩余存储空间多的 group 优先。

After a group is selected, the tracker picks a storage server within that group using round‑robin, IP order, or priority order.

当选定 group 后，tracker 会在 group 内选择一个 storage server 给客户端，支持如下选择 storage 的规则：1. Round robin，2. First server ordered by ip，3. First server ordered by priority（在 storage 上配置）。

The chosen storage then selects a data directory based on round‑robin or the directory with the most free space.

当分配好 storage server 后，客户端将向 storage 发送写文件请求，storage 将会为文件分配一个数据存储目录，支持如下规则：1. Round robin，2. 剩余存储空间最多的优先。

FastDFS generates a FileId by concatenating the storage IP, file creation time, size, CRC32, and a random number, then encoding the binary string with Base64.

选定存储目录之后，storage 会为文件生成一个 FileId，由 storage server ip、文件创建时间、文件大小、文件 crc32 和一个随机数拼接而成，然后将这个二进制串进行 base64 编码，转换为可打印的字符串。

Files are stored in a two‑level directory hierarchy (256 × 256). The FileId is hashed twice to determine the final sub‑directory, and the file is saved using the FileId as its name.

当选定存储目录之后，storage 会为文件分配一个 fileid，每个存储目录下有两级 256*256 的子目录，storage 会按文件 fileid 进行两次 hash（猜测），路由到其中一个子目录，然后将文件以 fileid 为文件名存储到该子目录下。

The final file name combines the group, storage path, two‑level sub‑directory, FileId, and the original file suffix.

当文件存储到某个子目录后，即认为该文件存储成功，接下来会为该文件生成一个文件名，文件名由 group、存储目录、两级子目录、fileid、文件后缀名（由客户端指定）拼接而成。

File synchronization – After a write succeeds on one storage, a background thread copies the file to the other storages in the same group. Each storage writes a binlog (containing only metadata) to track synchronization progress, which is reported to the tracker for read‑side decision making.

Download file – The client uses the file name returned by the upload to request the file. The tracker parses the file name to obtain group and other metadata, then selects a storage that is likely to have the file based on several rules (source storage alive, synchronization timestamps, etc.).

1. 该文件上传到的源头 storage 只要存活着，肯定包含这个文件，源头的地址被编码在文件名中。2. 文件创建时间戳 == storage 被同步到的时间戳 且 (当前时间‑文件创建时间戳) > 文件同步最大时间（如 5 分钟）‑文件创建后，认为经过最大同步时间后，肯定已经同步到其他 storage 了。3. 文件创建时间戳 < storage 被同步到的时间戳‑同步时间戳之前的文件确定已经同步了。4. (当前时间‑文件创建时间戳) > 同步延迟阀值（如一天）‑经过同步延迟阈值时间，认为文件肯定已经同步了。

Small‑file merging – To address inode limits, directory overhead, and backup inefficiency, FastDFS V3.0 introduces trunk files that store many small files together. The FileId is extended by 16 bytes to include trunk file ID, offset, and allocated size.

1. trunk file id 2. 文件在 trunk file 内的 offset 3. 文件占用的存储空间大小（字节对齐及删除空间复用，文件占用存储空间 >= 文件大小）

Each trunk file is created by a trunk server (selected by the tracker) and replicated across the group. Offsets are encoded in the file name; deleted space can be reused but only by files that fit the freed region.

HTTP access – Both tracker and storage embed HTTP support; the tracker redirects HTTP requests to the appropriate storage. Additional modules for Apache or Nginx can also be used.

Other features – FastDFS offers set/get meta‑data interfaces that store key‑value pairs in hidden meta files alongside the original file, and an appender file type that allows appending data after creation (but cannot be merged into trunk files).

Discussion of limitations – The design emphasizes simplicity, which leads to issues such as data safety (single‑copy writes can lose data if the source storage fails before sync), lack of automated recovery, low recovery efficiency, no multi‑datacenter disaster recovery, limited inode‑based storage capacity, space waste due to block allocation, and static load‑balancing based on groups.

Overall, FastDFS provides a straightforward solution for distributed storage of medium‑sized files, but its simplicity brings challenges in reliability, scalability, and efficient space utilization.