How to Build a Scalable Chunked File Upload Service with Java and Redis

Solution Overview

When handling large files, the usual approach is to split the file into chunks, upload each chunk separately, and finally merge them on the server.

Server‑Side Design

The server uses Redis to store a multipart upload session (file ID, name, size, status) and a map of chunk ETag values. Two REST endpoints are provided: one to initialize the upload and obtain an uploadId, one to receive each chunk together with its partNumber and ETag, and a third to complete the upload by merging all chunks with FileChannel.transferFrom(). An abort endpoint removes the temporary files and Redis entries.

try (FileChannel finalChannel = new RandomAccessFile(finalFile, "rw").getChannel()) {
    partsMap.keySet().stream().sorted().forEach(partNumber -> {
        File partFile = new File(SHARED_STORAGE_DIR, uploadId + "_part_" + partNumber);
        try (FileChannel partChannel = new RandomAccessFile(partFile, "r").getChannel()) {
            finalChannel.transferFrom(partChannel, finalChannel.size(), partChannel.size());
            partFile.delete();
        } catch (IOException e) {
            throw new RuntimeException("Chunk merge failed", e);
        }
    });
}

After merging, the session status is set to COMPLETE and the chunk map is deleted.

Progress Monitoring

Each successful chunk upload increments a counter stored in Redis; a separate API can query this counter to return the upload progress as a percentage.

Client‑Side Implementation

The client splits the local file into 10 MB parts using RandomAccessFile and FileChannel, writes each part to a temporary file, and sends it to the server via HTTP. A thread pool executes the uploads in parallel.

try (RandomAccessFile raf = new RandomAccessFile(filePath, "r");
     FileChannel fileChannel = raf.getChannel()) {
    ByteBuffer buffer = ByteBuffer.allocate((int) size);
    fileChannel.position(offset);
    fileChannel.read(buffer);
    buffer.flip();
    byte[] data = buffer.array();
    File temp = new File(TEMP_DIR, uploadId + "_part_" + partNumber);
    new FileOutputStream(temp).write(data);
}

After all parts are uploaded, the client calls the /complete endpoint with the list of PartETag objects.

Resume‑Supported Upload

A checkpoint file (named by the MD5 of the source file) records the uploadId and the ETag of each successfully uploaded chunk. Before uploading, the client loads the checkpoint; already uploaded chunks are skipped, and new chunks are appended to the checkpoint after each successful upload.

private CheckpointData loadCheckpoint(String fileName, Long fileSize, File checkpointFile) {
    if (!checkpointFile.exists()) {
        String uploadId = initiateUpload(fileName, fileSize);
        saveCheckpoint(checkpointFile, "{\"uploadId\":\"" + uploadId + "\"}");
        return new CheckpointData(uploadId, new HashMap<>());
    }
    // read existing uploadId and partETag map from file
}

Shared Storage in a Distributed Environment

To make the upload service work across multiple Docker nodes, an NFS share is mounted on each host (e.g., /mnt/shared-storage) and then bound into the container as /shared/storage. All nodes read and write the same physical directory, allowing the server to access uploaded chunks regardless of the node that received them.

sudo mount -t nfs <nfs_server_ip>:/export/shared-storage /mnt/shared-storage

docker run -v /mnt/shared-storage:/shared/storage your_image

With this architecture, large files can be uploaded reliably, monitored in real time, resumed after failures, and processed in a horizontally scalable Docker cluster.