Backend Development 17 min read
Implementing Large File Upload and Download with Chunking, Resume, and Multi‑Threaded Support Using WebUploader and Spring Boot
This article demonstrates how to build a robust large‑file upload and download solution in Java by integrating Baidu's WebUploader on the front end with Spring Boot back‑end APIs that handle chunked uploads, breakpoint resume, file merging, and multi‑threaded range‑based downloads.
Top Architect
Top Architect
WebUploader, developed by Baidu's WebFE team, is a modern file‑upload component that primarily uses HTML5 with optional Flash support. It supports features such as chunked (slice) uploading, breakpoint resume, and instant (秒传) upload based on file MD5.
Implementation idea :
Split the file into slices according to a custom buffer size.
For each slice, generate a unique name and upload it; if a slice already exists on the server, skip it (breakpoint resume).
After all slices are uploaded, merge them in order to reconstruct the original file.
Instant upload checks a stored MD5 hash and avoids re‑uploading identical files.
Backend setup : Create a Spring Boot project and add the following Maven dependencies:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
</dependencies>Verify the environment with a simple test controller:
@Controller
public class UploadController {
@RequestMapping("/up")
@ResponseBody
public String upload(HttpServletRequest request, HttpServletResponse response) {
return "搭建成功";
}
}The main page contains the upload UI and the JavaScript that configures WebUploader:
<body>
<div id="upload-container">上传</div>
<div id="upload-list"></div>
<button id="picker">点击上传</button>
</body>
<script>
$('#upload-container').click(function(){
$("#picker").find('input').click();
});
var uploader = WebUploader.create({
auto: true,
swf: 'Uploader.swf',
server: 'http://localhost:8080/upload',
dnd: '#upload-container',
pick: '#picker',
multiple: true,
chunked: true,
threads: 20,
method: 'POST',
fileSizeLimit: 1024*1024*1024*10,
fileSingleSizeLimit: 1024*1024*1024,
fileVal: 'upload'
});
// event listeners for logging and UI updates omitted for brevity
</script>Server‑side chunked upload handling (UploadController):
@Controller
public class UploadController {
private static final String UTF8 = "utf-8";
@RequestMapping("/up")
@ResponseBody
public void upload(HttpServletRequest request, HttpServletResponse response) throws Exception {
response.setCharacterEncoding(UTF8);
Integer chunk = null; // current slice index
Integer chunks = null; // total number of slices
String name = null; // original file name
String path = "D:\\file"; // temporary storage directory
BufferedOutputStream os = null;
try {
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setSizeThreshold(1024);
factory.setRepository(new File(path));
ServletFileUpload upload = new ServletFileUpload(factory);
upload.setFileSizeMax(5L*1024*1024*1024);
upload.setSizeMax(10L*1024*1024*1024);
List
items = upload.parseRequest(request);
for (FileItem item : items) {
if (item.isFormField()) {
if ("chunk".equals(item.getFieldName())) {
chunk = Integer.parseInt(item.getString(UTF8));
}
if ("chunks".equals(item.getFieldName())) {
chunks = Integer.parseInt(item.getString(UTF8));
}
if ("name".equals(item.getFieldName())) {
name = item.getString(UTF8);
}
}
}
for (FileItem item : items) {
if (!item.isFormField()) {
String tempFileName = name;
if (name != null && chunk != null) {
tempFileName = chunk + "_" + name;
}
File tempFile = new File(path, tempFileName);
if (!tempFile.exists()) {
item.write(tempFile);
}
}
}
// merge when the last slice arrives
if (chunk != null && chunk.intValue() == chunks.intValue() - 1) {
File finalFile = new File(path, name);
os = new BufferedOutputStream(new FileOutputStream(finalFile));
for (int i = 0; i < chunks; i++) {
File part = new File(path, i + "_" + name);
while (!part.exists()) {
Thread.sleep(100);
}
byte[] bytes = FileUtils.readFileToByteArray(part);
os.write(bytes);
os.flush();
part.delete();
}
os.flush();
}
response.getWriter().write("上传成功");
} finally {
if (os != null) {
os.close();
}
}
}
}Server‑side range (slice) download (DownLoadController):
@Controller
public class DownLoadController {
private static final String UTF8 = "utf-8";
@RequestMapping("/down")
public void downLoadFile(HttpServletRequest request, HttpServletResponse response) throws IOException {
response.setCharacterEncoding(UTF8);
File file = new File("D:\\File\\a.mp4");
long fSize = file.length();
response.setContentType("application/x-download");
String fileName = URLEncoder.encode(file.getName(), UTF8);
response.addHeader("Content-Disposition", "attachment;filename=" + fileName);
response.setHeader("Accept-Range", "bytes");
response.setHeader("fSize", String.valueOf(fSize));
response.setHeader("fName", fileName);
long pos = 0, last = fSize - 1, sum = 0;
if (request.getHeader("Range") != null) {
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
String range = request.getHeader("Range").replaceAll("bytes=", "");
String[] parts = range.split("-");
if (parts.length == 2) {
pos = Long.parseLong(parts[0].trim());
last = Long.parseLong(parts[1].trim());
if (last > fSize - 1) last = fSize - 1;
} else {
pos = Long.parseLong(range.replaceAll("-", "").trim());
}
}
long rangeLength = last - pos + 1;
response.setHeader("Content-Range", "bytes" + pos + "-" + last + "/" + fSize);
response.setHeader("Content-Lenght", String.valueOf(rangeLength));
try (BufferedOutputStream os = new BufferedOutputStream(response.getOutputStream());
BufferedInputStream is = new BufferedInputStream(new FileInputStream(file))) {
is.skip(pos);
byte[] buffer = new byte[1024];
int len;
while (sum < rangeLength) {
len = is.read(buffer, 0, (int)Math.min(buffer.length, rangeLength - sum));
sum += len;
os.write(buffer, 0, len);
}
}
}
}Client‑side multi‑threaded slice download (DownloadClient):
@RestController
public class DownloadClient {
private static final long PER_PAGE = 1024L*1024L*50L;
private static final String DOWN_PATH = "D:\\File";
ExecutorService pool = Executors.newFixedThreadPool(10);
@RequestMapping("/downloadFile")
public String downloadFile() throws IOException {
FileInfo fileInfo = download(0, 10, -1, null);
if (fileInfo != null) {
long pages = fileInfo.fSize / PER_PAGE;
for (int i = 0; i <= pages; i++) {
pool.submit(new Download(i*PER_PAGE, (i+1)*PER_PAGE-1, i, fileInfo.fName));
}
}
return "成功";
}
class Download implements Runnable {
long start, end, page; String fName;
Download(long start, long end, long page, String fName) {
this.start = start; this.end = end; this.page = page; this.fName = fName;
}
@Override
public void run() {
try { download(start, end, page, fName); } catch (IOException e) { e.printStackTrace(); }
}
}
private FileInfo download(long start, long end, long page, String fName) throws IOException {
File file = new File(DOWN_PATH, page + "-" + fName);
if (file.exists() && page != -1 && file.length() == PER_PAGE) return null;
HttpClient client = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://127.0.0.1:8080/down");
httpGet.setHeader("Range", "bytes=" + start + "-" + end);
HttpResponse response = client.execute(httpGet);
String fSize = response.getFirstHeader("fSize").getValue();
fName = URLDecoder.decode(response.getFirstHeader("fName").getValue(), "utf-8");
HttpEntity entity = response.getEntity();
try (InputStream is = entity.getContent();
FileOutputStream fos = new FileOutputStream(file)) {
byte[] buffer = new byte[1024];
int ch;
while ((ch = is.read(buffer)) != -1) {
fos.write(buffer, 0, ch);
}
}
if (end - Long.valueOf(fSize) > 0) {
mergeFile(fName, page);
}
return new FileInfo(Long.valueOf(fSize), fName);
}
private void mergeFile(String fName, long page) throws Exception {
File finalFile = new File(DOWN_PATH, fName);
try (BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(finalFile))) {
for (int i = 0; i <= page; i++) {
File part = new File(DOWN_PATH, i + "-" + fName);
while (!part.exists() || (i != page && part.length() < PER_PAGE)) {
Thread.sleep(100);
}
byte[] bytes = FileUtils.readFileToByteArray(part);
os.write(bytes);
os.flush();
part.delete();
}
}
}
class FileInfo { long fSize; String fName; FileInfo(long fSize, String fName) { this.fSize = fSize; this.fName = fName; } }
}The article concludes with a call for discussion, promotional links to other technical articles, and a QR‑code invitation to join a "Top‑Level Architect" community, but the core instructional content remains the end‑to‑end solution for large file transfer using chunking, resume, and parallel download techniques.
Written by
Top Architect
Top Architect focuses on sharing practical architecture knowledge, covering enterprise, system, website, large‑scale distributed, and high‑availability architectures, plus architecture adjustments using internet technologies. We welcome idea‑driven, sharing‑oriented architects to exchange and learn together.
0 followers
Reader feedback
How this landed with the community
Rate this article
Was this worth your time?
Discussion
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.