How to Efficiently Upload Large Files with Chunking, Resume, and Instant Transfer

What is a large file

Generally, a large file refers to files larger than 100 MB, while ordinary files are smaller than 100 MB, commonly 20 MB, 30 MB, or 50 MB. The main differences are size and transmission speed. Email attachments usually allow only 20–50 MB, making transfer of several hundred megabytes difficult.

Differences between uploading large and ordinary files

Uploading ordinary files only requires two points: specify the upload endpoint and set the request header Content‑Type to multipart/form‑data, sending the file as a binary stream.

Uploading large files encounters problems such as request timeout and size limits, network jitter causing retries, HTTP/1.1 head‑of‑line blocking, and lack of progress indication.

Principles and ideas of large‑file upload

Frontend

The frontend reads the file’s binary content, splits it into slices of a specified size, and uploads each slice to the server.

Key optimizations: breakpoint resume, instant transfer (秒传), and progress display.

Backend

The backend receives slice files identified by a unique identifier, reassembles them into the original file.

Key optimization: delete slice files after merging.

When restoring slices, ensure each slice can be associated with its file and handle out‑of‑order arrival.

Implementation

Frontend slicing code

// Get identifier, same file returns same value
function createIdentifier(file) {
    return file.name + file.size;
}
let file = document.querySelector("[name=file]").files[0];
const LENGTH = 1024 * 1024 * 1; // 1 MB
let chunks = slice(file, LENGTH);
let identifier = createIdentifier(file);
let tasks = [];
chunks.forEach((chunk, index) => {
    let fd = new FormData();
    fd.append("file", chunk);
    fd.append("identifier", identifier);
    fd.append("chunkNumber", index + 1);
    fd.append("totalChunks", chunks.length);
    tasks.push(post("/mkblk.php", fd));
});
Promise.all(tasks).then(res => {
    let fd = new FormData();
    fd.append("identifier", identifier);
    fd.append("totalChunks", chunks.length);
    post("/mkfile.php", fd).then(res => {
        console.log(res);
    });
});

Backend slice merging code (PHP)

// mkblk.php
$identifier = $_POST['identifier'];
$path = './upload/' . $identifier;
if (!is_dir($path)) {
    mkdir($path);
}
$filename = $path . '/' . $_POST['chunkNumber'];
move_uploaded_file($_FILES['file']['tmp_name'], $filename);

// mkfile.php
$identifier = $_POST['identifier'];
$totalChunks = (int)$_POST['totalChunks'];
$filename = "./upload/{$identifier}/file.jpg";
for ($i = 1; $i <= $totalChunks; ++$i) {
    $file = "./upload/{$identifier}/{$i}";
    $content = file_get_contents($file);
    $fd = file_exists($filename) ? fopen($filename, "a") : fopen($filename, "w+");
    fwrite($fd, $content);
}

Further optimizations include breakpoint resume, instant transfer, progress monitoring, and pause/resume using xhr.upload progress events and xhr.abort.

Existing mature solutions

Libraries such as Qiniu SDK, Tencent Cloud SDK, and the Vue component vue-simple-uploader provide chunked upload, pause/resume, instant transfer, and progress features.

Conclusion

The article introduced the definition of large files, differences in uploading, the chunked upload principle, a simple implementation, and recommended the vue-simple-uploader component for practical use.