How to Achieve Near‑Instant npm Installations: Benchmarks and Optimizations

Table of Contents

Performance Comparison: benchmark results showing installation speed.

Optimization Methods: the journey of improvements and lessons learned.

Development Path: current status and future direction.

Performance Comparison

Benchmark uses the internal npm package @alipay/smallfish with over 2,000 dependencies, transferring 38 MB of data (over 400 MB after extraction). Tests run on an M1 Mac mini (8 C/16 GB) over a gigabit wired network without any npm cache. The registry is an internal private source to ensure consistent latency.

Installation times:

npm: 1 min 07 s

Yarn (optimized): 36 s

PNPM / CNPM (soft‑link optimization): 19 s

npm rapid : 6 s (≈10× faster than npm)

Optimization Methods

Aggregation

npm performs two HTTP requests per package (metadata and tarball). By aggregating the 2,000+ requests into a single request to a custom service that runs @npmcli/arborist and caches results in memory, distributed cache, or DB, the dependency‑tree generation time drops from 15 s to 3 s, a reduction of 80 %.

Middle Layer

IO overhead dominates installation: npm makes ~200,000 write syscalls (≈76 µs each) during a typical install. To cut this, a middle‑layer intercepts file operations and writes packages as tar archives instead of many small files. By merging two tar files via fs.appendFile, IO calls drop from 26 to 3, reducing total install time from ~30 s to ~3 s (≈90 % reduction).

Tar archives, however, are not directly usable by Node's require. To preserve the existing developer workflow, a FUSE‑based file system (NPM FS) is introduced, exposing tar contents as regular files.

Cache (Overlay & COW)

After speeding up installation, disk usage remains a problem. npm already caches tar files globally, but repeated extraction still consumes time and space. By leveraging OverlayFS with Copy‑On‑Write (COW), a single global cache can serve multiple projects while keeping project‑specific modifications isolated, eliminating the “npm black‑hole” of excessive disk consumption.

Development Path

npm has energized the JavaScript ecosystem; the team contributed cnpm as a domestic mirror to accelerate installs. The next steps are to open‑source NPM FS, revert to npm’s original directory layout (removing soft‑link incompatibilities), and add full npm workspace support so that the rapid mode can eventually be merged upstream, benefiting all JavaScript developers.

Interested contributors can contact the presenter via the contact information shown in the slides.