Mastering Large File Uploads: Instant, Chunked, and Resumable Strategies in Java

Introduction

File upload for large files cannot rely on simple byte‑stream upload because interruptions force a restart; better user experience requires more advanced techniques.

Instant (Hash‑Based) Upload

What is instant upload

Server checks the MD5 of the incoming file; if the same file already exists, it returns a reference URL, avoiding re‑upload. Changing the file content (not just the name) changes the MD5 and disables instant upload.

Core logic

Use Redis set to store upload status keyed by file MD5; the value indicates whether the upload is complete. If true, subsequent uploads trigger the instant path; if false, store the chunk file path with a prefixed key.

Chunked (Slice) Upload

Definition

Split the file into equal‑size parts (slices) and upload each separately; after all parts arrive the server merges them into the original file.

Scenarios

Large file upload

Unstable network where retransmission risk exists

Resumable Upload

What is resumable upload

Divide the file into multiple parts, each handled by a thread; if network failure occurs, upload can continue from the last successful part instead of restarting.

Application scenario

Same as chunked upload; resumable upload is a derivative of chunked upload.

Core logic

During chunked upload, if the process crashes, the client records progress; the server provides an interface to query already uploaded slices so the client can resume from the next slice.

Implementation steps

Two approaches are described:

Conventional steps: split file, initialize an upload task, send slices (serial or parallel), server assembles when all slices are received.

Implementation used in this article: client sends slice index and size; server creates a .conf file whose length equals total slice count; each uploaded slice writes a marker byte (127) at its position; unuploaded slices remain 0. When all bytes are 127, the upload is complete.

Backend Code Implementation

The front‑end uses Baidu WebUploader for slicing (see

http://fex.baidu.com/webuploader/getting-started.html

).

RandomAccessFile strategy

@UploadMode(mode = UploadModeEnum.RANDOM_ACCESS)
@Slf4j
public class RandomAccessUploadStrategy extends SliceUploadTemplate {
    @Autowired
    private FilePathUtil filePathUtil;

    @Value("${upload.chunkSize}")
    private long defaultChunkSize;

    @Override
    public boolean upload(FileUploadRequestDTO param) {
        RandomAccessFile accessTmpFile = null;
        try {
            String uploadDirPath = filePathUtil.getPath(param);
            File tmpFile = super.createTmpFile(param);
            accessTmpFile = new RandomAccessFile(tmpFile, "rw");
            // This must match the front‑end setting
            long chunkSize = Objects.isNull(param.getChunkSize()) ? defaultChunkSize * 1024 * 1024 : param.getChunkSize();
            long offset = chunkSize * param.getChunk();
            // Position to the slice offset
            accessTmpFile.seek(offset);
            // Write the slice data
            accessTmpFile.write(param.getFile().getBytes());
            boolean isOk = super.checkAndSetUploadProgress(param, uploadDirPath);
            return isOk;
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        } finally {
            FileUtil.close(accessTmpFile);
        }
        return false;
    }
}

MappedByteBuffer strategy

@UploadMode(mode = UploadModeEnum.MAPPED_BYTEBUFFER)
@Slf4j
public class MappedByteBufferUploadStrategy extends SliceUploadTemplate {
    @Autowired
    private FilePathUtil filePathUtil;

    @Value("${upload.chunkSize}")
    private long defaultChunkSize;

    @Override
    public boolean upload(FileUploadRequestDTO param) {
        RandomAccessFile tempRaf = null;
        FileChannel fileChannel = null;
        MappedByteBuffer mappedByteBuffer = null;
        try {
            String uploadDirPath = filePathUtil.getPath(param);
            File tmpFile = super.createTmpFile(param);
            tempRaf = new RandomAccessFile(tmpFile, "rw");
            fileChannel = tempRaf.getChannel();

            long chunkSize = Objects.isNull(param.getChunkSize()) ? defaultChunkSize * 1024 * 1024 : param.getChunkSize();
            // Write the slice data
            long offset = chunkSize * param.getChunk();
            byte[] fileData = param.getFile().getBytes();
            mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, offset, fileData.length);
            mappedByteBuffer.put(fileData);
            boolean isOk = super.checkAndSetUploadProgress(param, uploadDirPath);
            return isOk;
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        } finally {
            FileUtil.freedMappedByteBuffer(mappedByteBuffer);
            FileUtil.close(fileChannel);
            FileUtil.close(tempRaf);
        }
        return false;
    }
}

A common abstract template SliceUploadTemplate defines the upload workflow, progress tracking, Redis persistence, and file renaming. (The full template code is omitted for brevity.)

Key Points and Recommendations

Front‑end and back‑end must agree on slice size and index; otherwise the upload fails. A dedicated file server (e.g., FastDFS, HDFS) is usually required. In a test environment (4‑core CPU, 8 GB RAM) uploading a 24 GB file took about 30 minutes, with most time spent on client‑side MD5 calculation; back‑end write speed remains fast.

For projects that only need upload/download without building a custom file server, using a cloud object storage such as Alibaba OSS is recommended (see

https://help.aliyun.com/product/31815.html

). OSS is an object storage service, not a file system, so it may not suit scenarios with heavy delete or modify operations. An OSS form‑upload demo is available at

https://www.cnblogs.com/ossteam/p/4942227.html

.