How to Efficiently Upload GB‑Scale Files with Spring Boot

1. Introduction

Uploading large files (over 100 MB) with a single request often leads to network instability, excessive server memory usage, and costly retries. The article proposes a chunked upload solution to overcome these three shortcomings.

2. Why Chunked Upload?

Reduces load of each request.

Supports resumable uploads.

Allows concurrent transmission for higher overall throughput.

Alleviates server memory pressure.

3. Chunked Upload Core Principle

The client splits a large file into many small pieces, sends each piece to a temporary directory on the server, and after all pieces arrive the server merges them into the original file, dramatically lowering long‑connection and high‑memory risks.

4. Practical Implementation

4.1 Controller API Definition

<pre><code>@RestController
@RequestMapping("/upload")
public class ChunkUploadController {
    private final String CHUNK_DIR = "uploads/chunks/";
    private final String FINAL_DIR = "uploads/final/";

    // Initialize upload
    @PostMapping("/init")
    public ResponseEntity<String> initUpload(@RequestParam String fileName,
                                            @RequestParam String fileMd5) {
        String uploadId = UUID.randomUUID().toString();
        Path chunkDir = Paths.get(CHUNK_DIR, fileMd5 + "_" + uploadId);
        try {
            Files.createDirectories(chunkDir);
        } catch (IOException e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                                 .body("Failed to create directory");
        }
        return ResponseEntity.ok(uploadId);
    }

    // Upload a chunk
    @PostMapping("/chunk")
    public ResponseEntity<String> uploadChunk(@RequestParam MultipartFile chunk,
                                            @RequestParam String uploadId,
                                            @RequestParam String fileMd5,
                                            @RequestParam Integer index) {
        String chunkName = "chunk_" + index + ".tmp";
        Path filePath = Paths.get(CHUNK_DIR, fileMd5 + "_" + uploadId, chunkName);
        try {
            chunk.transferTo(filePath);
            return ResponseEntity.ok("Chunk uploaded successfully");
        } catch (IOException e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                                 .body("Failed to save chunk");
        }
    }

    // Merge all chunks
    @PostMapping("/merge")
    public ResponseEntity<String> mergeChunks(@RequestParam String fileName,
                                            @RequestParam String uploadId,
                                            @RequestParam String fileMd5) {
        File chunkDir = new File(CHUNK_DIR + fileMd5 + "_" + uploadId);
        File[] chunks = chunkDir.listFiles();
        if (chunks == null || chunks.length == 0) {
            return ResponseEntity.badRequest().body("No chunk files found");
        }
        Arrays.sort(chunks, Comparator.comparingInt(f ->
            Integer.parseInt(f.getName().split("_")[1].split("\\.")[0])));
        Path finalPath = Paths.get(FINAL_DIR, fileName);
        try (BufferedOutputStream outputStream =
                 new BufferedOutputStream(Files.newOutputStream(finalPath))) {
            for (File chunkFile : chunks) {
                Files.copy(chunkFile.toPath(), outputStream);
            }
            FileUtils.deleteDirectory(chunkDir);
            return ResponseEntity.ok("File merged successfully: " + finalPath);
        } catch (IOException e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                                 .body("Merge failed: " + e.getMessage());
        }
    }
}
</code></pre>

4.2 High‑Performance Merging for >10 GB Files

<pre><code>public void mergeFiles(File targetFile, List<File> chunkFiles) throws IOException {
    try (RandomAccessFile target = new RandomAccessFile(targetFile, "rw")) {
        byte[] buffer = new byte[1024 * 8]; // 8 KB buffer
        long position = 0;
        for (File chunk : chunkFiles) {
            try (RandomAccessFile src = new RandomAccessFile(chunk, "r")) {
                int bytesRead;
                while ((bytesRead = src.read(buffer)) != -1) {
                    target.write(buffer, 0, bytesRead);
                }
                position += chunk.length();
            }
        }
    }
}
</code></pre>

4.3 Front‑End Chunk Handling

<pre><code>// 5 MB per chunk
const CHUNK_SIZE = 5 * 1024 * 1024;

function processFile(file) {
    const chunkCount = Math.ceil(file.size / CHUNK_SIZE);
    const chunks = [];
    for (let i = 0; i < chunkCount; i++) {
        const start = i * CHUNK_SIZE;
        const end = Math.min(file.size, start + CHUNK_SIZE);
        chunks.push(file.slice(start, end));
    }
    return chunks;
}

async function uploadFile(file) {
    const { data: uploadId } = await axios.post('/upload/init', {
        fileName: file.name,
        fileMd5: await calculateFileMD5(file)
    });
    const chunks = processFile(file);
    const total = chunks.length;
    let uploaded = 0;
    await Promise.all(chunks.map((chunk, index) => {
        const formData = new FormData();
        formData.append('chunk', chunk, `chunk_${index}`);
        formData.append('index', index);
        formData.append('uploadId', uploadId);
        formData.append('fileMd5', fileMd5);
        return axios.post('/upload/chunk', formData, {
            headers: { 'Content-Type': 'multipart/form-data' },
            onUploadProgress: progress => {
                const percent = ((uploaded * 100) / total).toFixed(1);
                updateProgress(percent);
            }
        }).then(() => uploaded++);
    }));
    const result = await axios.post('/upload/merge', {
        fileName: file.name,
        uploadId,
        fileMd5
    });
    alert(`上传成功: ${result.data}`);
}
</code></pre>

4.4 Resumable Upload Check

<pre><code>@GetMapping("/check/{fileMd5}/{uploadId}")
public ResponseEntity<List<Integer>> getUploadedChunks(@PathVariable String fileMd5,
                                                    @PathVariable String uploadId) {
    Path chunkDir = Paths.get(CHUNK_DIR, fileMd5 + "_" + uploadId);
    if (!Files.exists(chunkDir)) {
        return ResponseEntity.ok(Collections.emptyList());
    }
    try {
        List<Integer> uploaded = Files.list(chunkDir)
            .map(p -> p.getFileName().toString())
            .filter(name -> name.startsWith("chunk_"))
            .map(name -> name.replace("chunk_", "").replace(".tmp", ""))
            .map(Integer::parseInt)
            .collect(Collectors.toList());
        return ResponseEntity.ok(uploaded);
    } catch (IOException e) {
        return ResponseEntity.status(500).body(Collections.emptyList());
    }
}
</code></pre>

On the client side the returned list is used to skip already uploaded chunks before sending the remaining ones.

4.5 Data‑Block Integrity with HMAC‑SHA256

<pre><code>@PostMapping("/chunk")
public ResponseEntity<?> uploadChunk(@RequestParam MultipartFile chunk,
                                    @RequestParam String sign) {
    String secretKey = "your-secret-key";
    String serverSign = HmacUtils.hmacSha256Hex(secretKey, chunk.getBytes());
    if (!serverSign.equals(sign)) {
        return ResponseEntity.status(403).body("签名验证失败");
    }
    // process chunk …
}
</code></pre>

4.6 Optional Cloud Storage with MinIO

<pre><code>@Configuration
public class MinioConfig {
    @Bean
    public MinioClient minioClient() {
        return MinioClient.builder()
                         .endpoint("http://minio:9000")
                         .credentials("minio-access", "minio-secret")
                         .build();
    }
}

@Service
public class MinioUploadService {
    private final MinioClient minioClient;
    public void uploadChunk(String bucket, String object,
                            InputStream chunkStream, long length) throws Exception {
        minioClient.putObject(PutObjectArgs.builder()
            .bucket(bucket)
            .object(object)
            .stream(chunkStream, length, -1)
            .build());
    }
}
</code></pre>

The article concludes that using chunked uploads, resumable checks, secure HMAC verification, and optional MinIO storage together provide a robust solution for GB‑level file uploads in Spring Boot 3.5.0.