Efficient Large File Uploads: Chunking, Instant Transfer & Resume in Java

Introduction

File upload is a common problem. For small files a simple byte‑stream upload works, but for large files the traditional approach leads to poor user experience because an interruption forces a full restart. This article introduces several techniques that provide a smoother upload experience.

Detailed Techniques

Instant Transfer (秒传)

What is instant transfer?

When a file is uploaded, the server first calculates its MD5. If a file with the same MD5 already exists, the server returns a new URL pointing to the existing file, avoiding a real upload. Changing the MD5 (e.g., by modifying the file content) disables instant transfer.

Core logic of instant transfer in this article

a) Use Redis SET to store the upload status, with the MD5 as the key and a flag indicating completion as the value.

b) If the flag is true, a duplicate file triggers instant transfer. If false, the server stores the path of each chunk file using a key composed of the MD5 and a fixed prefix.

Chunked (Slice) Upload

What is chunked upload?

Chunked upload splits a large file into multiple parts of a fixed size, uploads each part separately, and then the server merges the parts back into the original file.

Scenarios for chunked upload

Large file upload

Unstable network where retransmission risk exists

Breakpoint Resume

What is breakpoint resume?

Breakpoint resume divides a file into several segments, each handled by a separate thread. If a network failure occurs, the upload can continue from the last successfully uploaded segment instead of restarting from the beginning. This article focuses on breakpoint upload for the upload scenario.

Application scenarios

Breakpoint resume is essentially a derivative of chunked upload, so it can be used in all scenarios where chunked upload applies.

Core logic of breakpoint resume

During chunked upload, if the system crashes or the network disconnects, the client records the upload progress. When the upload is retried, the client continues from the last recorded position.

The server can provide an interface for the client to query already uploaded chunks, allowing the client to start from the next chunk.

Implementation steps

a) Conventional steps

Split the file into equal‑size chunks according to a fixed rule.

Initialize a chunked upload task and return a unique identifier.

Send each chunk according to a chosen strategy (serial or parallel).

After all chunks are sent, the server verifies completeness and merges the chunks into the original file.

b) Steps used in this article

The front‑end splits the file into fixed‑size chunks and includes the chunk index and size in each request to the back‑end.

The back‑end creates a .conf file to record chunk status; each byte in the file represents a chunk (127 for uploaded, 0 for not uploaded). This file is the core of both breakpoint resume and instant transfer.

The server calculates the offset based on the chunk index and size, reads the incoming data, and writes it to the target file.

Code implementation of chunked / breakpoint upload

Front‑end uses Baidu's WebUploader plugin for chunking. The back‑end provides two file‑write strategies.

http://fex.baidu.com/webuploader/getting-started.html

Back‑end file write implementations

a) RandomAccessFile implementation

@UploadMode(mode = UploadModeEnum.RANDOM_ACCESS)
@Slf4j
public class RandomAccessUploadStrategy extends SliceUploadTemplate {
    @Autowired
    private FilePathUtil filePathUtil;

    @Value("${upload.chunkSize}")
    private long defaultChunkSize;

    @Override
    public boolean upload(FileUploadRequestDTO param) {
        RandomAccessFile accessTmpFile = null;
        try {
            String uploadDirPath = filePathUtil.getPath(param);
            File tmpFile = super.createTmpFile(param);
            accessTmpFile = new RandomAccessFile(tmpFile, "rw");
            long chunkSize = Objects.isNull(param.getChunkSize()) ? defaultChunkSize * 1024 * 1024 : param.getChunkSize();
            long offset = chunkSize * param.getChunk();
            accessTmpFile.seek(offset);
            accessTmpFile.write(param.getFile().getBytes());
            boolean isOk = super.checkAndSetUploadProgress(param, uploadDirPath);
            return isOk;
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        } finally {
            FileUtil.close(accessTmpFile);
        }
        return false;
    }
}

b) MappedByteBuffer implementation

@UploadMode(mode = UploadModeEnum.MAPPED_BYTEBUFFER)
@Slf4j
public class MappedByteBufferUploadStrategy extends SliceUploadTemplate {
    @Autowired
    private FilePathUtil filePathUtil;

    @Value("${upload.chunkSize}")
    private long defaultChunkSize;

    @Override
    public boolean upload(FileUploadRequestDTO param) {
        RandomAccessFile tempRaf = null;
        FileChannel fileChannel = null;
        MappedByteBuffer mappedByteBuffer = null;
        try {
            String uploadDirPath = filePathUtil.getPath(param);
            File tmpFile = super.createTmpFile(param);
            tempRaf = new RandomAccessFile(tmpFile, "rw");
            fileChannel = tempRaf.getChannel();
            long chunkSize = Objects.isNull(param.getChunkSize()) ? defaultChunkSize * 1024 * 1024 : param.getChunkSize();
            long offset = chunkSize * param.getChunk();
            byte[] fileData = param.getFile().getBytes();
            mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, offset, fileData.length);
            mappedByteBuffer.put(fileData);
            boolean isOk = super.checkAndSetUploadProgress(param, uploadDirPath);
            return isOk;
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        } finally {
            FileUtil.freedMappedByteBuffer(mappedByteBuffer);
            FileUtil.close(fileChannel);
            FileUtil.close(tempRaf);
        }
        return false;
    }
}

c) Core template class for file operations

@Slf4j
public abstract class SliceUploadTemplate implements SliceUploadStrategy {
    public abstract boolean upload(FileUploadRequestDTO param);

    protected File createTmpFile(FileUploadRequestDTO param) {
        FilePathUtil filePathUtil = SpringContextHolder.getBean(FilePathUtil.class);
        param.setPath(FileUtil.withoutHeadAndTailDiagonal(param.getPath()));
        String fileName = param.getFile().getOriginalFilename();
        String uploadDirPath = filePathUtil.getPath(param);
        String tempFileName = fileName + "_tmp";
        File tmpDir = new File(uploadDirPath);
        if (!tmpDir.exists()) {
            tmpDir.mkdirs();
        }
        return new File(uploadDirPath, tempFileName);
    }

    @Override
    public FileUploadDTO sliceUpload(FileUploadRequestDTO param) {
        boolean isOk = this.upload(param);
        if (isOk) {
            File tmpFile = this.createTmpFile(param);
            return this.saveAndFileUploadDTO(param.getFile().getOriginalFilename(), tmpFile);
        }
        String md5 = FileMD5Util.getFileMD5(param.getFile());
        Map<Integer, String> map = new HashMap<>();
        map.put(param.getChunk(), md5);
        return FileUploadDTO.builder().chunkMd5Info(map).build();
    }

    public boolean checkAndSetUploadProgress(FileUploadRequestDTO param, String uploadDirPath) {
        String fileName = param.getFile().getOriginalFilename();
        File confFile = new File(uploadDirPath, fileName + ".conf");
        byte isComplete = 0;
        RandomAccessFile accessConfFile = null;
        try {
            accessConfFile = new RandomAccessFile(confFile, "rw");
            accessConfFile.setLength(param.getChunks());
            accessConfFile.seek(param.getChunk());
            accessConfFile.write(Byte.MAX_VALUE);
            byte[] completeList = FileUtils.readFileToByteArray(confFile);
            isComplete = Byte.MAX_VALUE;
            for (int i = 0; i < completeList.length && isComplete == Byte.MAX_VALUE; i++) {
                isComplete = (byte) (isComplete & completeList[i]);
            }
        } catch (IOException e) {
            log.error(e.getMessage(), e);
        } finally {
            FileUtil.close(accessConfFile);
        }
        return setUploadProgress2Redis(param, uploadDirPath, fileName, confFile, isComplete);
    }

    private boolean setUploadProgress2Redis(FileUploadRequestDTO param, String uploadDirPath, String fileName, File confFile, byte isComplete) {
        RedisUtil redisUtil = SpringContextHolder.getBean(RedisUtil.class);
        if (isComplete == Byte.MAX_VALUE) {
            redisUtil.hset(FileConstant.FILE_UPLOAD_STATUS, param.getMd5(), "true");
            redisUtil.del(FileConstant.FILE_MD5_KEY + param.getMd5());
            confFile.delete();
            return true;
        } else {
            if (!redisUtil.hHasKey(FileConstant.FILE_UPLOAD_STATUS, param.getMd5())) {
                redisUtil.hset(FileConstant.FILE_UPLOAD_STATUS, param.getMd5(), "false");
                redisUtil.set(FileConstant.FILE_MD5_KEY + param.getMd5(), uploadDirPath + FileConstant.FILE_SEPARATORCHAR + fileName + ".conf");
            }
            return false;
        }
    }

    public FileUploadDTO saveAndFileUploadDTO(String fileName, File tmpFile) {
        FileUploadDTO fileUploadDTO = null;
        try {
            fileUploadDTO = renameFile(tmpFile, fileName);
            if (fileUploadDTO.isUploadComplete()) {
                // TODO: persist file metadata to DB
            }
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
        return fileUploadDTO;
    }

    private FileUploadDTO renameFile(File toBeRenamed, String toFileNewName) {
        FileUploadDTO fileUploadDTO = new FileUploadDTO();
        if (!toBeRenamed.exists() || toBeRenamed.isDirectory()) {
            log.info("File does not exist: {}", toBeRenamed.getName());
            fileUploadDTO.setUploadComplete(false);
            return fileUploadDTO;
        }
        String ext = FileUtil.getExtension(toFileNewName);
        String p = toBeRenamed.getParent();
        String filePath = p + FileConstant.FILE_SEPARATORCHAR + toFileNewName;
        File newFile = new File(filePath);
        boolean uploadFlag = toBeRenamed.renameTo(newFile);
        fileUploadDTO.setMtime(DateUtil.getCurrentTimeStamp());
        fileUploadDTO.setUploadComplete(uploadFlag);
        fileUploadDTO.setPath(filePath);
        fileUploadDTO.setSize(newFile.length());
        fileUploadDTO.setFileExt(ext);
        fileUploadDTO.setFileId(toFileNewName);
        return fileUploadDTO;
    }
}

Conclusion

Implementing chunked upload requires coordination between front‑end and back‑end, especially consistent chunk size and index. A dedicated file server (e.g., FastDFS, HDFS) is typically needed. In a test environment (4‑core CPU, 8 GB RAM) uploading a 24 GB file took about 30 minutes, with most time spent on MD5 calculation on the client side.

If building a file server is too costly, using an object storage service such as Alibaba OSS is an alternative, though it is not ideal for scenarios requiring frequent deletions or modifications.

https://help.aliyun.com/product/31815.html

OSS also supports direct browser‑to‑OSS form uploads, offloading the upload traffic entirely to the storage service.

https://www.cnblogs.com/ossteam/p/4942227.html