Production-Grade Large File Upload in Spring Boot: Chunking, Resume & OSS Direct Transfer

1. Why Traditional Direct Upload Fails for Large Files

Spring Boot's default upload uses the Servlet container's temporary file mechanism. Once files exceed ~100 MB, off-heap memory triggers alerts. Even with spring.servlet.multipart.max-file-size configured to spill to disk, synchronous reading of the entire file still blocks Tomcat worker threads.

In public networks, connections drop unpredictably (Wi-Fi to 4G handoff, screen lock, carrier throttling). Failed uploads restart from zero, wasting bandwidth. Worse, the application server — meant for business logic — becomes a file relay gateway. When egress bandwidth saturates, core query APIs slow down, causing cascade failures. The path Client → App Server → Object Storage doubles traffic and I/O, an anti-pattern in microservice architectures.

The solution: split large files into independently transferable chunks; the server only handles routing and metadata; data flow bypasses the app server entirely.

2. Designing a Chunked Protocol Without Ambiguity

A workable chunk protocol must nail three things: task identification, transfer control, and merge ordering.

2.1 Core Field Conventions

{
  "uploadId": "biz_20231024_x9d2",
  "fileHash": "a1b2c3d4e5f6...",
  "fileName": "raw_video.mp4",
  "totalSize": 1048576000,
  "totalChunks": 200,
  "chunkIndex": 15,
  "chunkSize": 5242880
}

uploadId

is the global task identifier; chunkIndex starts at 0. totalChunks must match the frontend slicing logic exactly — both sides must compute identically.

2.2 To Hash or Not to Hash?

File-level hash (MD5 or SHA-256) is mandatory: the frontend sends it for instant deduplication (秒传), and the server re-verifies after merge to detect tampering. Chunk-level hash is optional; HTTPS and cloud storage CRC already provide integrity. Computing a digest per chunk doubles CPU cost — only enable for financial-grade requirements.

2.3 Concurrency and Out-of-Order Handling

Browser concurrent requests should stay at 3–5 active windows. More triggers gateway rate limits or cloud vendor 429 Too Many Requests; fewer underutilizes bandwidth. Out-of-order arrival is normal — e.g., index=18 arrives before index=5. During merge, sort strictly by chunkIndex; never by file modification time or receive time, or the reconstructed file will be corrupt.

3. Server-Side Reception and Async Merge (Production Code)

The server role must stay thin: the receive endpoint only persists chunks and logs metadata; merging runs off the HTTP request thread.

3.1 Chunk Reception Endpoint

@RestController
@RequestMapping("/api/v1/upload")
@RequiredArgsConstructor
public class ChunkUploadController {

    private final ChunkUploadService chunkService;

    @PostMapping("/chunk")
    public ResponseEntity<Void> uploadChunk(
            @RequestParam("file") MultipartFile chunk,
            @RequestParam String uploadId,
            @RequestParam int chunkIndex) {
        chunkService.saveChunk(uploadId, chunk, chunkIndex);
        return ResponseEntity.ok().build();
    }
}

3.2 Async Merge Logic

Merge must not run on the main thread — it would timeout gateway requests. Use a dedicated thread pool with FileChannel for zero-copy.

@Service
@Slf4j
@RequiredArgsConstructor
public class ChunkUploadService {

    private final String tempBaseDir = System.getProperty("java.io.tmpdir") + "/uploads";
    private final ThreadPoolTaskExecutor mergeExecutor;

    public void saveChunk(String uploadId, MultipartFile chunk, int index) {
        Path taskDir = Paths.get(tempBaseDir, uploadId);
        try {
            Files.createDirectories(taskDir);
            Path target = taskDir.resolve("chunk_" + index);
            // Use Files.copy instead of transferTo to avoid container temp-file cleanup races
            try (InputStream is = chunk.getInputStream()) {
                Files.copy(is, target, StandardCopyOption.REPLACE_EXISTING);
            }
            // Record uploaded chunk index in Redis Set, key: upload:chunks:{uploadId}
        } catch (IOException e) {
            throw new RuntimeException("Chunk save failed", e);
        }
    }

    @Async("mergeExecutor")
    public void mergeChunks(String uploadId, String fileName, int totalChunks) {
        Path taskDir = Paths.get(tempBaseDir, uploadId);
        try {
            List<Path> chunks = Files.list(taskDir)
                    .filter(p -> p.getFileName().toString().startsWith("chunk_"))
                    .sorted(Comparator.comparingInt(this::extractIndex))
                    .collect(Collectors.toList());

            if (chunks.size() != totalChunks) {
                log.warn("Chunk mismatch for {}: expected {}, got {}", uploadId, totalChunks, chunks.size());
                // Production should trigger retry or alert; throwing exception for illustration
                throw new IllegalStateException("Missing chunks");
            }

            Path finalDir = Paths.get("/data/files/final");
            Files.createDirectories(finalDir);
            Path merged = finalDir.resolve(fileName);

            // Zero-copy merge
            try (FileChannel out = FileChannel.open(merged,
                    StandardOpenOption.CREATE,
                    StandardOpenOption.WRITE,
                    StandardOpenOption.TRUNCATE_EXISTING)) {
                for (Path chunk : chunks) {
                    try (FileChannel in = FileChannel.open(chunk, StandardOpenOption.READ)) {
                        out.transferFrom(in, out.position(), in.size());
                    }
                }
            }

            // Cleanup temp task directory
            deleteRecursively(taskDir.toFile());
            log.info("Merge success: {}", fileName);

        } catch (Exception e) {
            log.error("Merge failed for uploadId: {}", uploadId, e);
            // Production: log failure and push to retry queue
        }
    }

    private int extractIndex(Path p) {
        String name = p.getFileName().toString();
        return Integer.parseInt(name.substring(name.indexOf("_") + 1));
    }
}

Key Practical Lessons: FileChannel.transferFrom is highly efficient on Linux, but watch the position accumulation. Using out.position() directly is safer than manual variable tracking.

Temp directories need scheduled cleanup. Run an XXL-JOB or Spring @Scheduled task scanning for orphan directories older than 24 hours, or disks fill up in days.

Merge failures need a fallback: retry up to 3 times, then mark MERGE_FAILED so the frontend can re-trigger merge or escalate to manual intervention.

4. Bypassing the App Server: OSS/MinIO Direct Upload

Once files hit GB scale, stop routing traffic through Spring Boot. The app server does only two things: issue credentials and record metadata. All data I/O goes to cloud storage.

4.1 STS Temporary Credentials vs. Presigned URLs

STS suits clients using the cloud vendor SDK directly — fine-grained permissions (bucket, object prefix, max size, even PutObject -only). Presigned URLs are lighter: the server computes a signed PUT URL and hands it to the frontend, which uses a standard HTTP request. Both work; choose based on frontend infrastructure. Presigned URL TTL should be 10–15 minutes — expire fast to prevent scraping.

@Service
public class OssPresignService {
    private final S3Client ossClient; // MinIO or AWS SDK compatible client

    public String generatePresignedUrl(String objectKey, int expireMinutes) {
        GeneratePresignedUrlRequest req = new GeneratePresignedUrlRequest(
                "your-bucket", objectKey);
        req.setExpiration(Date.from(Instant.now().plusMinutes(expireMinutes)));
        req.setMethod(HttpMethod.PUT);
        return ossClient.generatePresignedUrl(req).toString();
    }
}

Cloud providers deeply optimize multipart upload (CDN acceleration, global transfer, auto-merge). Wire up the InitiateMultipartUpload, UploadPart, CompleteMultipartUpload state machine and let infrastructure handle the rest.

5. Resumable Upload and Frontend Progress Integration

Resumable upload boils down to gap detection + state sync.

5.1 Backend Status Query

@GetMapping("/status")
public ResponseEntity<List<Integer>> getUploadedChunks(
        @RequestParam String fileHash) {
    Set<String> indices = redis.opsForSet().members(
            "upload:chunks:" + fileHash);
    return ResponseEntity.ok(indices.stream()
            .map(Integer::parseInt)
            .sorted()
            .toList());
}

Redis Set stores uploaded chunk indices. The frontend compares this list against total chunks and re-uploads only the missing ones.

5.2 Frontend Resume Logic (Watch the FormData Pitfall)

Many tutorials get this wrong. axios file uploads must use FormData; sending a raw JSON object fails parsing on cloud storage or backend.

async function resumeUpload(file, hash) {
    const { data: uploaded } = await axios.get(
        `/api/v1/upload/status?fileHash=${hash}`);
    const chunkSize = 5 * 1024 * 1024;
    const total = Math.ceil(file.size / chunkSize);
    const missing = [];

    for (let i = 0; i < total; i++) {
        if (!uploaded.includes(i)) missing.push(i);
    }

    const uploadPromises = missing.map(index => {
        const start = index * chunkSize;
        const end = Math.min(start + chunkSize, file.size);
        const chunkBlob = file.slice(start, end);

        const formData = new FormData();
        formData.append('file', chunkBlob);
        formData.append('uploadId', generateUploadId(hash));
        formData.append('chunkIndex', index);

        return axios.post(`/api/v1/upload/chunk`, formData, {
            headers: { 'Content-Type': 'multipart/form-data' },
            onUploadProgress: evt => updateProgress(index, evt.loaded, chunkSize)
        });
    });

    await Promise.all(uploadPromises);
    // Notify backend to trigger merge
}

Progress calculation stays simple:

(completedChunks * chunkSize + currentChunkBytes) / totalSize * 100

. Large-file hashing is CPU-intensive — never run on the main thread. Offload to a Web Worker with SparkMD5, reading the File object in chunks to keep the browser responsive.

6. Security Must Be Front-Loaded

Large-file pipelines attract scrapers and malicious uploads; without hardening, servers get stuffed.

6.1 Don't Trust Content-Type

Client-sent Content-Type and file extensions are trivially spoofed. Server must inspect the file's magic number. Apache Tika works, but don't stream the whole file — read the first 4 KB for detection. Block executables ( .php, .jsp, .sh). For images/video, also validate EXIF headers to catch steganographic payloads.

6.2 Access Control and Rate Limiting

Configure cloud storage Referer and IP allowlists to restrict access to business domains. Presigned URLs are single-use — no long-lived tokens. Rate-limit with Redis + Lua sliding window: e.g., max 20 chunk requests per user per minute. Combine with business-context quota checks to prevent bucket exhaustion. After upload, async-scan with ClamAV or cloud security center; quarantine and alert on malware hits.

7. Implementation Recommendations

Early file services used client → app server → local disk. Fine for internal tools; under load it OOMs and saturates bandwidth. Next evolution: server receives chunks, async merges, pushes to NFS/FTP — thread-pool tuning and temp-dir cleanup consumed half the team's life. Today production shifts to cloud direct upload. The app server becomes a control plane; data plane lives in OSS. Architecture looks simpler, but details get sharper: how to guarantee eventual metadata consistency? How to alert and retry on merge failure? How to refresh STS tokens transparently before expiry? These are the real bottlenecks.

Don't go all-in on chunking from day one. First, simulate network flakiness and disconnects; verify Redis state tracking and temp-file cleanup hold up. During canary rollout, watch core metrics: chunk backlog rate, merge failure rate, temp-directory disk watermark. Only when control plane and data plane are fully decoupled is large-file upload truly solved.