Implementing a Net‑Disk‑Style 10 GB File Upload with Spring Boot: No Restart After Interrupt

When uploading large files (e.g., 1 GB, 3 GB, 10 GB) with the usual MultipartFile approach, any brief network glitch causes the upload to stop at around 92% and restart from zero, making the experience unacceptable.

The proposed solution replaces the single‑request upload with three techniques: chunked upload, resumable upload, and instant upload.

Why the ordinary MultipartFile cannot handle big files

A typical Spring Boot endpoint looks like:

@PostMapping("/upload")
public String upload(@RequestParam MultipartFile file) throws IOException {
    file.transferTo(Path.of("/data/uploads/", file.getOriginalFilename()));
    return "success";
}

For a 10 GB file this means one long‑lasting HTTP request; any failure (Wi‑Fi drop, Nginx timeout, server restart, browser close) forces the whole upload to fail.

Chunking the file

Assume a 10 GB file and a 10 MB chunk size, yielding roughly 1024 chunks numbered from chunk 0 to chunk 1023. The browser now sends, for each chunk, the file hash, the chunk index, the total number of chunks, and the chunk data.

fileHash: 7d52a87d...
chunkIndex: 37
totalChunks: 1024

Thus the upload transforms from a single 10 GB request into 1024 separate 10 MB requests. If chunk 387 fails, only that chunk is retransmitted while the previously uploaded chunks remain intact.

Backend step 1 – giving each file a unique identity

Using only the original filename is unsafe because different users may upload files with the same name but different content. The frontend first computes a SHA‑256 hash of the file (e.g., 96e8b7c1a4...) and sends it to the server.

The server defines a request record:

public record ChunkRequest(String fileHash, Integer chunkIndex, Integer totalChunks, String fileName) {}

And an upload endpoint:

@PostMapping("/upload/chunk")
public void uploadChunk(@RequestParam String fileHash,
                       @RequestParam int chunkIndex,
                       @RequestParam int totalChunks,
                       @RequestParam String fileName,
                       @RequestParam MultipartFile chunk) throws IOException {
    uploadService.saveChunk(fileHash, chunkIndex, totalChunks, fileName, chunk);
}

Chunks are stored under a temporary directory per file hash:

/data/upload-temp/
    96e8b7c1a4/
        0.part
        1.part
        2.part
        ...

True resumable upload – querying server state

Clients can ask the server which chunks have already been received:

@GetMapping("/upload/status/{fileHash}")
public UploadStatus status(@PathVariable String fileHash) {
    return uploadService.getStatus(fileHash);
}

The response lists uploaded chunk indices, allowing the browser to skip them and continue from the next missing chunk.

Merging chunks after all are uploaded

When all 1024 chunks are present, the client calls POST /upload/complete. The server merges the parts using Java NIO streams:

public Path merge(String fileHash, int totalChunks, String fileName) throws IOException {
    Path chunkDir = Path.of("/data/upload-temp", fileHash);
    Path target = Path.of("/data/uploads", fileName);
    try (OutputStream output = new BufferedOutputStream(Files.newOutputStream(target))) {
        for (int i = 0; i < totalChunks; i++) {
            Path part = chunkDir.resolve(i + ".part");
            Files.copy(part, output);
        }
    }
    return target;
}

Importantly, the implementation streams each part instead of loading the whole 10 GB into memory, avoiding OutOfMemoryError.

Instant upload ("秒传")

Before sending any data, the client can request a quick check:

POST /upload/check
{ "fileHash": "96e8b7c1a4...", "fileSize": 10737418240 }

The server runs a query such as:

SELECT id, storage_path FROM file_object WHERE file_hash = ? AND file_size = ? LIMIT 1;

If a matching record exists, the server creates a reference for the current user without uploading the file content, achieving an instant upload.

Security considerations

The backend must not trust the client‑provided hash blindly. It should verify the hash, file size, user permissions, and file status, and after merging recompute the SHA‑256 to ensure integrity. File names must not be used directly as paths to prevent directory traversal and overwriting; instead, generate an internal file ID with a safe extension.

Concurrency limits and cleanup

Uploading all chunks concurrently can overwhelm the server (hundreds of simultaneous requests, random disk writes, CPU/IO spikes). Frontends typically limit to 3‑6 concurrent chunks. The backend should also enforce:

Maximum concurrent uploads per user (e.g., 5).

Maximum file size (e.g., 20 GB).

Automatic cleanup of temporary chunk directories that remain in UPLOADING state for over 24 hours.

Without such limits, disk space becomes the bottleneck.

Conclusion

Large‑file upload is not a simple POST operation; it requires a full system that handles file hashing, instant‑upload checks, chunk management, retry logic, resumable recovery, integrity verification, merging, record creation, and temporary data cleanup. Spring Boot excels at managing this stateful lifecycle.