Reliable MQTT File Transfer with FastDFS and Redis

Background

The existing IoT collection system communicates with devices via the MQTT protocol. Terminal devices usually operate on 3G/4G networks with limited data traffic. Besides uploading monitoring data, business scenarios often require file upload or download, with each transmission limited to a maximum of 1 KB. The system must also support resumable transfer when the network or CPU is idle. Implementing these requirements in a single‑process system is straightforward, but in a distributed architecture it becomes difficult because file requests and responses are not fixed, requiring a unified solution for packetization, transmission, and storage without duplicate uploads or wasted traffic.

Solution Purpose and Highlights

Purpose: Provide a stable, reliable, and resumable file transfer mechanism for distributed IoT platforms that supports both upload and download over MQTT.

Highlights: Combine the distributed file storage middleware FastDFS with Redis as a shared cache. Use the file’s MD5 hash as a unique identifier. Store packet sequence numbers, distributed storage paths, and offsets in Redis. Devices split files, serialize each packet to Base64, and send them via MQTT. The receiving server decodes the Base64, writes the bytes to FastDFS, and records the metadata in Redis, enabling seamless reconstruction.

Implementation Details

1. Platform → Terminal Device (Upload)

The business system submits a file to the file‑processing service.

The service uploads the file to FastDFS and saves the MD5 and FastDFS path in Redis, e.g., file:md5:upload:{md5} → FastDFS path.

The MQTT producer sends the first packet to the device. The device acknowledges with fields fileMd5, packSize, and packOffset.

If the device loses connection, it retries after recovery, sending the last packOffset so the platform can resume from that offset.

The MQTT consumer (clustered) receives the acknowledgment, looks up the FastDFS path via Redis, reads the file, and continues sending remaining packets based on packSize and packOffset. Progress is stored in Redis under file:md5:split:{md5} (e.g., 4/45).

2. Terminal Device → Platform (Download)

The platform sends a download request containing fileType, packOffset (starting from 0), and packSize.

The device responds with fileMd5, fileSize, packSize, packOffset, and packData (Base64‑encoded chunk).

Each MQTT consumer stores the received chunk in FastDFS and records the chunk path in a Redis hash keyed by file:md5:download:{md5}, where the hash field is the offset and the value is the FastDFS path.

After writing the chunk, the consumer updates the download progress in Redis ( file:md5:split:{md5}), e.g., 12/45.

When the final packet is received (detected by packOffset + packSize > fileSize), the platform retrieves all chunk paths from the Redis hash, sorts them by offset, reads each chunk from FastDFS, concatenates the byte arrays, and streams the complete file to the business system.

Key Architectural Components

FastDFS: Distributed file storage for persisting each packet.

Redis: Cache for MD5‑based keys, file paths, and transfer progress.

MQTT: Message protocol used for transmitting Base64‑encoded packets between platform and devices.