Skip to content
10GB File Upload: Chunked Concurrency, Resumable & Instant Upload with MinIO + Redis

10GB File Upload: Chunked Concurrency, Resumable & Instant Upload with MinIO + Redis

This article presents a complete solution for uploading 10GB files, covering frontend chunking with WebWorker MD5, concurrent upload control, backend MinIO multipart upload with Redis state tracking, resumable upload, instant upload via hash deduplication, Java NIO zero-copy merging, and interview-focused Q&A with code examples.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Architecture Overview

The solution splits a 10GB file into N small chunks, uploads them concurrently from the frontend, temporarily stores chunks on the backend, and merges them after all chunks arrive. It supports resumable upload and instant upload (秒传) to handle network interruptions and duplicate uploads.

Architecture overview
Architecture overview

Key Technical Points

1. Frontend Chunking and Concurrency Control

Chunk size: 5MB–20MB (too small increases HTTP requests, too large raises retry cost).

Concurrency limit: 3–5 concurrent uploads (browsers limit same-origin connections to ~6).

MD5 calculation optimization: Use SparkMD5 library for incremental MD5 computation, avoiding loading the entire file into memory and crashing the browser.

Chunk sequencing: Assign each chunk a sequence number (e.g., file_001, file_002) to ensure correct merge order.

2. Backend Chunk Reception and Merging

Three implementation approaches compared:

Local disk staging + merge: Simple, low cost; but single-machine I/O bottleneck, no distributed support. Suitable for small teams, single-node deployment.

MinIO/S3 multipart upload: Native support for chunking, merging, concurrency, distributed architecture; requires deploying object storage. Suitable for medium/large enterprises, production.

Redis chunk status tracking: High performance, supports distributed locks; data loss risk without persistence. Required for all scenarios.

Recommended: MinIO native multipart upload + Redis for upload status. MinIO is fully S3-compatible, provides built-in Multipart Upload API handling chunk reception, merging, and concurrency control with minimal development effort.

3. Resumable Upload Implementation

On interruption, frontend sends file MD5; backend returns list of already uploaded chunk numbers.

Frontend only uploads missing chunks, avoiding duplicate transfers.

Backend uses Redis Hash: key = file MD5, field = chunk number, value = upload status.

4. Instant Upload (秒传)

Before upload, compute file MD5 and query backend for existing file with same MD5.

If exists, return success immediately without data transfer.

In enterprise scenarios, instant upload rate can exceed 30%, greatly improving user experience.

Java Backend Core Code Example

// Chunk upload endpoint
@PostMapping("/upload/chunk")
public ResponseEntity<ChunkUploadResult> uploadChunk(
    @RequestParam String fileMd5,
    @RequestParam Integer chunkNumber,
    @RequestParam Integer totalChunks,
    @RequestParam MultipartFile chunk) {

// 1. Check if chunk already uploaded
if (redisTemplate.opsForHash().hasKey(fileMd5, chunkNumber.toString())) {
    return ResponseEntity.ok(ChunkUploadResult.success("Chunk already uploaded"));
}

// 2. Save chunk to MinIO
String chunkObjectName = String.format("chunks/%s/%d", fileMd5, chunkNumber);
minioClient.putObject(
    PutObjectArgs.builder()
        .bucket("file-upload")
        .object(chunkObjectName)
        .stream(chunk.getInputStream(), chunk.getSize(), -1)
        .build()
);

// 3. Record chunk upload status
redisTemplate.opsForHash().put(fileMd5, chunkNumber.toString(), "uploaded");

// 4. Check if all chunks uploaded
Long uploadedChunks = redisTemplate.opsForHash().size(fileMd5);
if (uploadedChunks == totalChunks) {
    // Async merge
    fileMergeService.mergeChunksAsync(fileMd5, totalChunks);
}

return ResponseEntity.ok(ChunkUploadResult.success("Chunk uploaded successfully"));
}

Exception Handling and Optimizations

Network timeout: Set reasonable timeout (30s–60s), auto-retry 3 times on failure.

Chunk loss: Verify MD5 of all chunks before merge; notify frontend to retransmit missing chunks.

Memory overflow: Use streaming processing during backend merge to avoid loading all chunks into memory.

Disk space shortage: Set chunk expiration (24 hours), auto-clean unmerged chunks.

Slow large file merge: Use hard links or zero-copy techniques to reduce I/O operations.

Performance Reference Data

Based on typical home broadband (100Mbps–500Mbps):

10GB, 10MB chunks, 4 concurrent: 10MB/s → ~17 minutes

10GB, 10MB chunks, 4 concurrent: 20MB/s → ~8.5 minutes

10GB, 10MB chunks, 4 concurrent: 50MB/s → ~3.5 minutes

Interview Q&A Anticipation

Q: Why not use FTP?

A: FTP has poor penetration (firewalls often block FTP ports) and lacks resumable/instant upload, resulting in poor UX.

Q: How to handle MD5 collisions?

A: Use SHA-256 instead, or combine MD5 with file size and filename for double verification, making collision probability near zero.

Q: How to support files >100GB?

A: Increase chunk size to 50MB–100MB, raise concurrency to 6–8; use distributed object storage cluster to avoid single-node bottlenecks.

Core Code Highlights

Frontend: WebWorker + SparkMD5 Incremental MD5

Technical highlight: No full file load into memory; WebWorker multi-threaded calculation doesn't block UI thread; supports 100GB+ files with memory under 50MB.

// Main thread: invoke WebWorker for MD5
function calculateFileMD5(file) {
    return new Promise((resolve) => {
        const worker = new Worker('/js/md5-worker.js');
        worker.postMessage({ file });
        worker.onmessage = (e) => {
            worker.terminate();
            resolve(e.data.md5);
        };
    });
}

// md5-worker.js: background incremental MD5
importScripts('https://cdn.jsdelivr.net/npm/[email protected]/spark-md5.min.js');

self.onmessage = async (e) => {
    const file = e.data.file;
    const chunkSize = 2 * 1024 * 1024; // 2MB slices
    const chunks = Math.ceil(file.size / chunkSize);
    const spark = new SparkMD5.ArrayBuffer();
    let currentChunk = 0;
    const fileReader = new FileReader();
    fileReader.onload = (e) => {
        spark.append(e.target.result);
        currentChunk++;
        if (currentChunk < chunks) {
            loadNext();
        } else {
            self.postMessage({ md5: spark.end() });
        }
    };
    function loadNext() {
        const start = currentChunk * chunkSize;
        const end = Math.min(start + chunkSize, file.size);
        fileReader.readAsArrayBuffer(file.slice(start, end));
    }
    loadNext();
};

Frontend: Concurrency Control + Exponential Backoff Retry

Technical highlight: Promise-based concurrency pool, dynamic concurrency adjustment; exponential backoff retry (max 3 attempts, intervals 1s/2s/4s) to avoid network congestion.

class ConcurrentUploader {
    constructor(concurrency=4) {
        this.concurrency = concurrency;
        this.running = 0;
        this.queue = [];
    }
    addTask(task) {
        this.queue.push(task);
        this.run();
    }
    run() {
        while (this.running < this.concurrency && this.queue.length > 0) {
            const task = this.queue.shift();
            this.running++;
            task()
                .catch((err) => {
                    // Exponential backoff retry: max 3 times, intervals 1s/2s/4s
                    if (task.retryCount === undefined) task.retryCount = 0;
                    if (task.retryCount < 3) {
                        task.retryCount++;
                        setTimeout(() => {
                            this.queue.unshift(task);
                            this.run();
                        }, Math.pow(2, task.retryCount) * 1000);
                    }
                })
                .finally(() => {
                    this.running--;
                    this.run();
                });
        }
    }
}

// Usage
const uploader = new ConcurrentUploader(4);
chunks.forEach((chunk, index) => {
    uploader.addTask(() => uploadChunk(fileMd5, index, chunk));
});

Backend: Chunk-Level MD5 Integrity Verification

Technical highlight: Verifies each chunk's MD5, not just whole file; only retransmits corrupted chunks.

@PostMapping("/upload/chunk")
public ResponseEntity<ChunkUploadResult> uploadChunk(
    @RequestParam String fileMd5,
    @RequestParam Integer chunkNumber,
    @RequestParam String chunkMd5, // chunk MD5 from frontend
    @RequestParam MultipartFile chunk) {

// 1. Atomic check if chunk already uploaded
String redisKey = "upload:status:" + fileMd5;
if (redisTemplate.opsForHash().hasKey(redisKey, chunkNumber.toString())) {
    return ResponseEntity.ok(ChunkUploadResult.success("Chunk already uploaded"));
}

// 2. Verify chunk MD5 integrity
String calculatedChunkMd5 = DigestUtils.md5Hex(chunk.getInputStream());
if (!calculatedChunkMd5.equals(chunkMd5)) {
    return ResponseEntity.badRequest().body(ChunkUploadResult.fail("Chunk corrupted, please retry"));
}

// 3. Save chunk to MinIO
String chunkObjectName = String.format("chunks/%s/%d", fileMd5, chunkNumber);
minioClient.putObject(
    PutObjectArgs.builder()
        .bucket("file-upload")
        .object(chunkObjectName)
        .stream(chunk.getInputStream(), chunk.getSize(), -1)
        .build()
);

// 4. Atomic record chunk status
redisTemplate.opsForHash().put(redisKey, chunkNumber.toString(), chunkMd5);
redisTemplate.expire(redisKey, 24, TimeUnit.HOURS); // 24h expiry

// 5. Check if all chunks uploaded
Long uploadedChunks = redisTemplate.opsForHash().size(redisKey);
if (uploadedChunks == totalChunks) {
    // Send message to trigger async merge
    rabbitTemplate.convertAndSend("file-merge-queue", fileMd5);
}

return ResponseEntity.ok(ChunkUploadResult.success("Chunk uploaded successfully"));
}

Backend: Distributed Lock + Async Message Merge

Technical highlight: RabbitMQ decouples merge operation asynchronously; Redisson distributed lock prevents multiple instances merging same file simultaneously.

@Component
@RabbitListener(queues="file-merge-queue")
public class FileMergeConsumer {
    @Autowired
    private RedissonClient redissonClient;
    @Autowired
    private MinioClient minioClient;

    @RabbitHandler
    public void mergeFile(String fileMd5) {
        RLock lock = redissonClient.getLock("upload:merge:" + fileMd5);
        try {
            // Try lock, wait max 10s, auto-release after 30min
            if (lock.tryLock(10, 1800, TimeUnit.SECONDS)) {
                // 1. Get all chunk info from Redis
                String redisKey = "upload:status:" + fileMd5;
                Map<Object, Object> chunkStatus = redisTemplate.opsForHash().entries(redisKey);
                int totalChunks = chunkStatus.size();

                // 2. Zero-copy merge (see below)
                String finalObjectName = "files/" + fileMd5;
                zeroCopyMerge(fileMd5, totalChunks, finalObjectName);

                // 3. Verify overall MD5
                String finalMd5 = DigestUtils.md5Hex(minioClient.getObject(
                    GetObjectArgs.builder().bucket("file-upload").object(finalObjectName).build()
                ));
                if (!finalMd5.equals(fileMd5)) {
                    throw new RuntimeException("File merge failed, MD5 mismatch");
                }

                // 4. Delete temporary chunks and Redis status
                minioClient.removeObjects(
                    RemoveObjectsArgs.builder()
                        .bucket("file-upload")
                        .objects(IntStream.range(0, totalChunks)
                            .mapToObj(i -> "chunks/" + fileMd5 + "/" + i)
                            .collect(Collectors.toList()))
                        .build()
                );
                redisTemplate.delete(redisKey);
            }
        } finally {
            if (lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}

Backend: NIO Zero-Copy Merge

Technical highlight: Uses Java NIO transferTo for zero-copy, avoiding multiple kernel/user space copies; merge speed improves 10x+, CPU usage drops 80%.

private void zeroCopyMerge(String fileMd5, int totalChunks, String finalObjectName) throws Exception {
    // 1. Create output stream for final file
    ObjectWriteResponse finalFile = minioClient.putObject(
        PutObjectArgs.builder()
            .bucket("file-upload")
            .object(finalObjectName)
            .stream(null, -1, 10 * 1024 * 1024) // 10MB buffer
            .build()
    );

    // 2. Merge chunks sequentially using zero-copy
    try (FileChannel outChannel = ((FileOutputStream) finalFile).getChannel()) {
        for (int i = 0; i < totalChunks; i++) {
            String chunkObjectName = "chunks/" + fileMd5 + "/" + i;
            try (InputStream in = minioClient.getObject(
                    GetObjectArgs.builder().bucket("file-upload").object(chunkObjectName).build());
                 FileChannel inChannel = ((FileInputStream) in).getChannel()) {

                // Zero-copy: transfer directly from input channel to output channel
                inChannel.transferTo(0, inChannel.size(), outChannel);
            }
        }
    }
}

Technical Challenges and Solutions

Large file MD5 slow, browser crashes: Root cause: loading entire file into memory, JS single-thread blocking. Solution: Frontend WebWorker + SparkMD5 incremental calculation. Highlight: Memory <50MB, supports 100GB+, no UI blocking.

High-concurrency chunk upload state inconsistency: Root cause: distributed environment, multiple instances modifying same file state. Solution: Redis Hash atomic ops + Redisson distributed lock. Highlight: Ensures eventual consistency, prevents duplicate upload/merge.

Large file merge I/O bottleneck: Root cause: traditional byte copy requires 4 kernel/user switches. Solution: NIO zero-copy (transferTo) + MinIO native multipart merge. Highlight: Merge speed 10x+, CPU down 80%.

Network instability causing chunk corruption/loss: Root cause: data tampering or interruption during transfer. Solution: Chunk-level MD5 verification + exponential backoff auto-retry. Highlight: Only retransmit corrupted chunks, not whole file.

Browser same-origin connection limit: Root cause: Chrome/Firefox limit ~6 concurrent requests per domain. Solution: Dynamic concurrency adjustment + domain sharding (e.g., upload1.xxx.com). Highlight: Concurrency up to 12–18, upload speed doubles.

Expired chunks consuming disk space: Root cause: incomplete uploads leave orphan chunks. Solution: Redis expiry keys + scheduled cleanup job. Highlight: Auto-clean 24h unfinished chunks, free storage.

Inaccurate upload progress: Root cause: only counting completed chunks, not intra-chunk progress. Solution: Frontend listens to XMLHttpRequest progress events. Highlight: Real-time byte-level progress display.

Cross-origin issues: Root cause: browser same-origin policy. Solution: Backend CORS config + preflight request caching. Highlight: Supports cross-domain upload, reduces preflight requests.

Ultimate Performance Optimizations

CDN accelerated upload: Use Alibaba Cloud OSS / Tencent Cloud COS upload acceleration nodes to solve cross-region slowness.

P2P transfer: For enterprise internal scenarios, use WebRTC for P2P transfer, achieving gigabit speeds.

Persistent resumable upload: Save upload state to database for cross-browser, cross-device resumable upload.

Adaptive chunk sizing: Dynamically adjust chunk size based on user network speed (20MB for fast, 5MB for slow).

Real Interview Simulation

Interviewer asks candidate to design a 10GB file upload solution. Candidate outlines three pillars: chunked upload, resumable upload, instant upload; plus multi-threaded concurrency and direct-to-object-storage upload. Detailed discussion covers:

Instant upload and resumable upload via file hash fingerprint (SHA-256/MD5) computed in WebWorker; backend checks hash, returns URL if exists, else returns uploadId and missing chunk list.

Redis stores upload state keyed by uploadId with 24h expiry.

Client direct upload to OSS via presigned URLs (10-min validity); client collects ETags, calls merge endpoint.

Sequence diagram illustrates flow.

Concurrency control: Promise pool, 5–6 concurrent chunks considering HTTP/1.1 limit (6), HTTP/2 allows slightly more; chunk size ~5MB balances request duration and merge count (OSS requires min 5MB except last part).

Backend APIs: POST /api/file/initUpload (returns uploadId + missingChunks), POST /api/file/completeUpload (calls OSS CompleteMultipartUpload). Merge is synchronous for 10GB (seconds), can be async via message queue for higher loads.

Production considerations: file verification (hash comparison), token security (short-lived presigned URLs, permission checks), precise progress (WebSocket/SSE polling OSS), orphan chunk cleanup (scheduled AbortMultipartUpload), WebWorker compatibility (Safari fallback).

Sequence diagram
Sequence diagram

Key Scoring Points Recap

Chunking + resumable + instant upload → UX foundation

Direct OSS presigned URLs → bandwidth/concurrency silver bullet

WebWorker hash + Promise pool → frontend engineering highlight

Redis state management + presigned on-demand → backend architecture closure

Concurrency, cleanup, security details → proves production readiness

Core Code Demonstrations

Frontend: WebWorker SHA-256 + Concurrent Upload Pool

// 1. Web Worker computes SHA-256 (non-blocking)
// hash.worker.js
self.onmessage = async (e) => {
    const file = e.data;
    const chunkSize = 5 * 1024 * 1024; // 5MB chunks
    const chunks = Math.ceil(file.size / chunkSize);
    const hashBuffer = await crypto.subtle.digest(
        'SHA-256',
        await file.slice(0, chunkSize).arrayBuffer()
    );
    // Production would incrementally hash all chunks; simplified here
    const hashHex = Array.from(new Uint8Array(hashBuffer))
        .map(b => b.toString(16).padStart(2, '0')).join('');
    self.postMessage({ hash: hashHex, totalChunks: chunks });
};

// 2. Concurrent upload pool (max 6)
async function uploadFile(file) {
    const { hash, totalChunks } = await computeHashInWorker(file);
    const { uploadId, missingChunks } = await initUpload({ fileHash: hash, totalSize: file.size });
    if (missingChunks.length === 0) return { url: uploadId.url }; // instant upload
    const pool = new PromisePool(6);
    const partResults = [];
    for (const partNum of missingChunks) {
        const task = uploadPart(file, partNum, uploadId);
        pool.add(task.then(etag => partResults.push({ partNum, etag })));
    }
    await pool.drain();
    return await completeUpload({ uploadId, parts: partResults });
}

Highlight: WebWorker keeps hash off UI thread; Promise pool caps concurrency, prevents browser connection exhaustion.

Backend: Presigned URL Generation + Redis Resumable Logic

@PostMapping("/api/file/initUpload")
public InitResult initUpload(@RequestBody InitRequest req) {
    String hash = req.getFileHash();
    long size = req.getTotalSize();

    // 1. Instant upload detection: check if complete object exists
    if (ossClient.doesObjectExist(BUCKET, hash)) {
        return InitResult.quick(ossClient.generatePresignedUrl(BUCKET, hash, Duration.ofHours(1)));
    }

    // 2. Resumable: fetch existing upload record from Redis
    String uploadId = redisTemplate.opsForValue().get("upload:hash:" + hash);
    if (uploadId == null) {
        // First upload, create new multipart upload in OSS
        uploadId = ossClient.initiateMultipartUpload(BUCKET, hash).getUploadId();
    }

    // 3. Compute missing chunks
    Set<Integer> doneParts = redisTemplate.opsForSet()
        .members("upload:parts:" + uploadId).stream()
        .map(Integer::parseInt).collect(Collectors.toSet());
    List<Integer> missingChunks = IntStream.rangeClosed(1, (int) Math.ceil(size / (double) CHUNK_SIZE))
        .filter(i -> !doneParts.contains(i))
        .boxed().collect(Collectors.toList());

    // 4. Batch generate presigned URLs for missing chunks
    List<PartUploadUrl> urls = new ArrayList<>();
    for (int partNum : missingChunks) {
        String url = ossClient.generatePresignedUrl(BUCKET, hash, partNum, uploadId, Duration.ofMinutes(10));
        urls.add(new PartUploadUrl(partNum, url));
    }

    // 5. Write Redis, set 24h expiry
    redisTemplate.opsForValue().set("upload:hash:" + hash, uploadId, 24, TimeUnit.HOURS);
    return InitResult.resume(uploadId, urls);
}

Highlight: Hash-to-uploadId mapping + uploaded parts Set enables "same file multi-user shared progress"; presigned URLs issued on-demand, secure and resource-efficient.

Final Challenge Summary Table

1. Network jitter causing full retransmit: Chunked upload + resumable. Each chunk independent; file hash + Redis records uploaded chunks; refresh resumes from breakpoint. → "Divide and conquer, resume on break."

2. Browser memory/main thread blocked by hash: WebWorker background SHA-256, postMessage to main thread. → "Hash off UI, experience stays online."

3. Business server bandwidth saturated: Client direct-to-OSS, backend only signs presigned URLs, no file streaming. → "Traffic bypass, main pipeline zero load."

4. Duplicate files waste resources: Instant upload: pre-check hash against object storage, return link if exists. → "Upload once, duplicates instant."

5. Multi-chunk concurrency exhausts browser connections: Promise concurrency pool limits simultaneous chunks (e.g., 6), with HTTP/2 multiplexing. → "Valve control, speed and stability."

6. Interrupted uploads leave orphan chunks costing storage: Scheduled job scans expired uploadIds in Redis, calls OSS AbortMultipartUpload. → "Clean up, cost not wasted."

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

JavaRedisMinIOchunked uploadresumable uploadWebWorkerlarge file uploadinstant uploadconcurrent uploadNIO zero-copy
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.