How to Install npm Packages in Seconds: Inside Ant Group’s tnpm Rapid Mode

Background

As an experienced frontend engineer, I have felt the growing complexity of frontend development and the painfully slow speed of dependency installation.

On January 8, Ant Group frontend engineer Zero Yi delivered a keynote at SEE Conf 2022 titled “A Black‑Tech for Second‑Level npm Installation”. This article explains the background, thinking, results, and future of speeding up frontend dependency installation.

Why Is npm Slow?

In the modern npm ecosystem, the number of modules and their dependency relationships have become increasingly complex:

By the end of 2021, the npm registry contained over 1.8 million packages, far more than other language ecosystems.

Dependencies are tangled, with many duplicate and tiny files that waste disk space and slow write operations.

These factors cause npm installations to be very slow.

Installation Process

The typical installation workflow consists of three key steps:

Query sub‑dependency package information to obtain download URLs.

Download the .tgz archive and extract it locally.

Construct the node_modules directory structure and write files.

Dependency Example

Using [email protected] as a representative case, the project has roughly 1,000 direct dependencies, occupies 170 MB on disk, and contains 18,542 files.

When installed with npm 2 using the traditional nested layout, the effective dependency count rises to 3,626, with over 2,000 duplicate packages, 523 MB disk usage, and 60,257 files.

File I/O, especially the handling of massive numbers of tiny files, becomes a major bottleneck.

Flattening (npm 3) and Its Issues

npm 3 introduced a “flattened dependency” strategy, moving all sub‑dependencies to the root node_modules directory to reduce duplication and depth.

However, this approach introduced new problems:

Phantom dependencies (ghost packages).

Doppelganger dependencies – duplicate packages that remain (e.g., 183 duplicates).

Uncertain dependency structure – can be resolved with a dependency graph.

Increased algorithmic complexity and performance overhead.

pnpm’s Symlink + Hardlink Solution

pnpm solves the duplication issue by using a combination of soft links and hard links, which:

Eliminates duplicate packages while preserving Node.js resolution logic, avoiding phantom and doppelganger problems.

Uses a global cache with hard links to reduce file copying and save disk space.

Ant Group’s cnpm/npminstall adopted a similar approach, though without hard links and without promoting sub‑dependencies to the same level.

Potential Drawbacks of Symlinks

IDE indexing loops caused by symlinks (now largely mitigated).

Sub‑dependency promotion may require adaptation for tools like Egg or Webpack that rely on relative paths.

Hard‑linked files shared across projects can cause unintended side effects during debugging.

Hard links cannot cross file‑system boundaries; symlink implementations vary across OSes, and on non‑SSD disks they still incur I/O overhead.

Yarn’s Plug’n’Play was mentioned but not explored due to compatibility concerns.

Package Information Queries

Each dependency requires one package‑info request and one .tgz download, i.e., two HTTP requests. With thousands of dependencies, the number of HTTP requests scales up dramatically (npm 2 can issue over 2,500 requests).

Current optimization consensus: pre‑compute a dependency graph so that .tgz files can be downloaded directly without querying package metadata, cutting network latency by more than half.

Lockfiles (shrinkwrap, Yarn lock, pnpm lock) were originally intended for version locking but also serve as dependency graphs for faster downloads. Their drawbacks include lack of speed boost on first install and governance overhead in large‑scale use.

Optimization Results

To improve installation speed, Ant Group launched a dedicated effort in 2021, achieving a three‑fold speed increase and winning the Ant Group Luban Award.

The following sections dissect the optimization ideas behind the tnpm rapid mode.

Network I/O Optimizations

Server‑side Dependency Graph : Generate the graph on the server and apply a multi‑level cache, using @npmcli/arborist to follow npm specifications.

HTTP Pre‑heating : Warm up the registry and CDN endpoints before bulk download, reducing latency and avoiding intermittent DNS delays.

File Merging : Instead of extracting each .tgz individually, keep the archives as .tar files and merge them into about 40 tar bundles, drastically reducing write operations.

Rust‑based Downloader : Re‑implement download and extraction in Rust, running 40 concurrent coroutines for streaming download, extraction, and tar merging, which outperforms Node.js implementations.

FUSE Filesystem

We introduced a user‑space filesystem (FUSE) similar to a ServiceWorker for HTTP caching. The npmfs daemon registers as a FUSE service, mounts a virtual directory, and serves file reads by consulting the pre‑computed dependency graph and global cache.

This approach provides:

Transparent file access as if the directory were real.

Isolation of files to avoid hard‑link side effects.

OverlayFS Integration

To support temporary modifications of node_modules during development, we layer the FUSE mount as the lower directory of an OverlayFS, creating a writable upper layer. This copy‑on‑write mechanism enables safe, independent file changes while still sharing the underlying cache.

File I/O Optimizations

Global Cache Strategies : npm extracts .tgz to tar archives; pnpm extracts to files with hash‑based hard links. Both approaches still generate many small files.

Skipping Extraction : By mapping node_modules directly to tar archives via FUSE, we eliminate the extraction step, saving massive I/O.

Tar Merging : Consolidate thousands of .tgz into a handful of tar files, reducing disk write counts.

Additional Costs

FUSE compatibility across operating systems and container environments.

Server‑side dependency‑graph generation requires a private registry deployment; public mirrors fall back to client‑side generation.

Core Takeaways

Network I/O : Server‑generated dependency graphs eliminate per‑package metadata requests; Rust‑based download improves throughput.

File I/O : Merging writes into tar bundles and using FUSE to avoid extraction dramatically cuts disk operations.

Compatibility : Maintains the standard Node.js directory layout without the side effects of symlinks or flattening.

Future Outlook

Frontend package management has evolved over a decade, from npm’s early days to Yarn, and now pnpm. We aim to continue collaborating with the community, open‑source our innovations (e.g., npmfs, tnpm rapid), and promote standardization of package‑manager behavior.

About the Author

I am “Tian Zhu”, working in Ant Group’s Experience Technology Department in Guangzhou, focusing on frontend infrastructure. I contribute to open‑source projects such as Egg.js and cnpm, and I welcome collaborators.