Spring Boot 5GB File Upload: Chunked, Resumable & Instant Upload Implementation

Problem: Single-Request Upload Fails for Large Files

The original implementation used a single @PostMapping("/upload") with MultipartFile.transferTo() and max-file-size: 200MB. This worked for files up to tens of MB, but when users started uploading 800MB videos, 1.6GB installers, 3GB model files, and 5GB data files, the approach broke: a network hiccup at 96% forced a full restart, wasting bandwidth and user time.

Solution: Chunked Upload with Resumable and Instant Upload

The new flow:

Initialize upload (POST /api/uploads/init) with file name, size, SHA-256 hash, and total chunk count.

Client splits file into 8MB chunks (5GB ≈ 640 chunks).

Upload chunks concurrently (PUT /api/uploads/{uploadId}/chunks/{chunkNo}).

Server records each completed chunk in database.

On failure, client queries completed chunks (GET /api/uploads/{uploadId}) and resumes only missing ones.

When all chunks uploaded, client calls POST /api/uploads/{uploadId}/complete.

Server verifies chunk count, merges chunks via streaming, validates size/hash, atomically moves final file, updates status, cleans temp chunks.

Database Schema

Two tables: upload_task: id (PK), file_name, file_hash, file_size, total_chunks, status (UPLOADING/MERGING/COMPLETED/EXPIRED), final_path, timestamps, index on file_hash. upload_chunk: upload_id, chunk_no (composite PK), chunk_size, chunk_hash, created_at. Composite PK ensures idempotency: duplicate chunk uploads for same uploadId+chunkNo are rejected by DB constraint.

Initialization Logic (Instant Upload & Resume)

@Transactional
public InitUploadResponse init(InitUploadRequest request) {
    Optional completed = taskRepository.findCompletedByHash(request.fileHash(), request.fileSize());
    if (completed.isPresent()) {
        return new InitUploadResponse(completed.get().getId(), true, Set.of()); // instant upload
    }
    Optional uploading = taskRepository.findUploadingByHash(request.fileHash(), request.fileSize());
    if (uploading.isPresent()) {
        String uploadId = uploading.get().getId();
        Set chunks = chunkRepository.findChunkNumbers(uploadId);
        return new InitUploadResponse(uploadId, false, chunks); // resume
    }
    String uploadId = UUID.randomUUID().toString();
    UploadTask task = new UploadTask(uploadId, request.fileName(), request.fileHash(), request.fileSize(), request.totalChunks(), "UPLOADING");
    taskRepository.save(task);
    return new InitUploadResponse(uploadId, false, Set.of());
}

Instant upload means "not uploading at all" — server reuses existing completed file. Resume returns already uploaded chunk numbers so client skips them.

Chunk Upload: Streaming to Disk, Not Memory

Each chunk is 8MB. Using file.getBytes() would load entire chunk into heap; with 100 users × 4 concurrent chunks × 8MB = 3.2GB pressure. Instead, stream directly:

try (InputStream input = file.getInputStream()) {
    Files.copy(input, target, StandardCopyOption.REPLACE_EXISTING);
}

Spring Boot multipart limits can stay small (e.g., 12MB/16MB) because each request carries only one 8MB chunk.

Merge: Streaming Append, No Heap Bloat

Merge reads each chunk file sequentially and writes to a .merging file via BufferedOutputStream : <code>try (OutputStream output = new BufferedOutputStream(Files.newOutputStream(merging))) { for (int i = 0; i < task.getTotalChunks(); i++) { Path chunk = uploadDir.resolve(i + ".part"); if (!Files.exists(chunk)) throw new IllegalStateException("missing chunk: " + i); Files.copy(chunk, output); } }</code> Whether final file is 500MB or 20GB, JVM heap stays stable — Java only reads a buffer, writes a buffer. Validation & Atomic Finalization Check merged file size matches declared fileSize ; if mismatch, delete and throw. Optionally recompute SHA-256 and compare with client-provided hash. Atomically move .merging to final path using Files.move(..., ATOMIC_MOVE) . Update task status: UPLOADING → MERGING → COMPLETED. Delete temporary chunk directory. Concurrency Control: Database-Level Lock Multiple complete requests (e.g., client retry) could trigger parallel merges. Instead of JVM synchronized (fails across multiple Spring Boot instances), use atomic DB update: <code>UPDATE upload_task SET status = 'MERGING', updated_at = NOW() WHERE id = ? AND status = 'UPLOADING';</code> Only one request succeeds (rows=1); others get rows=0 and exit. Works across clustered deployments. Security: Tenant-Isolated Instant Upload Instant upload must not leak file existence across tenants. Query includes tenant_id: <code>WHERE tenant_id = ? AND file_hash = ? AND status = 'COMPLETED'</code> Prevents attackers from probing hashes to discover files they shouldn't access. Additional Hardening Filename sanitization : Never trust client filename for disk path. Store as UUID + extension; keep original name only as metadata. Expired upload cleanup : Scheduled job (cron 0 0 3 * * ?) deletes chunks and marks tasks EXPIRED if UPLOADING > 24 hours. Idempotent chunk upload : Composite PK (upload_id, chunk_no) guarantees exactly-once recording even if client retries due to missing response. Reflection: From Simple Request to Mini-Transaction The old model: "upload file = one HTTP request." New model: create task + send N retryable chunks + record each chunk state + submit task + verify + merge once + cleanup. Each chunk idempotent, task has state machine, failure recoverable, final result validated, merge serialized, expired tasks cleaned. Complexity is justified by production realities: network drops at 97%, duplicate chunk retries, double complete calls, multi-node deployments, 5GB files not blowing JVM heap. The author concludes: don't just increase max-file-size ; chunking solves the real problem — user never restarts from zero.