How Cloud‑Native Pipelines Cut Build Time 3‑5× with Remote Cache

Background

In cloud‑native pipelines each step runs in a fresh container, so code, dependencies and compiled artifacts are re‑downloaded and rebuilt on every run, leading to long execution times and wasted resources.

Goal

Implement a caching mechanism that reuses compiled artifacts across pipeline runs, achieving an average 3‑5× speedup.

Implementation

Cache files are compressed with Zstandard (zstd) and stored on a remote CFS mount. On subsequent builds the system checks for a cache hit; if found, it pulls and decompresses the cache, otherwise it builds and pushes the new cache.

Tools – CFS + Zstd

Base images include the required tools. Users can also use custom images and configure Restore and Save cache atoms with custom keys and directories.

1. CFS Remote Mount

Tools and startup scripts are baked into the base image, and the CFS client is launched at the start of the cache step to mount the remote filesystem.

_, err = c.ScriptAction.Sh([]string{ "sh", "-c", "modprobe fuse;cd /export/servers/tools/cfs;sudo ./cfs-client-randomwrite -c fuse.json" })

2. Zstd Compression

Zstandard, an open‑source lossless compressor from Facebook, offers a good balance of speed and compression ratio, outperforming tar, gzip, lz4, and xz. It is now supported by many big‑data tools such as Hadoop, Spark, and Kafka.

Cache Workflow

Check cache hit using a unique cache key derived from the project repository or user‑defined value.

If hit, pull the compressed cache from CFS and decompress to the appropriate directory.

If miss, after the build completes, compress the cache and push it to CFS.

storage.Watch("cache/",
    func(id string) { /* do nothing */ },
    func(id string) { CleanCache(id) })

Cache Limits and Eviction

The total cache size is limited by the allocated CFS quota (e.g., 20 GB). Caches not accessed for 7 days are automatically removed using etcd watch mechanisms and lease expiration.

Best Practices for Different Stacks

Java (Maven)

Cache the ${M2_HOME}/.m2 directory. Without cache the build takes ~5.26 min; with cache it drops to ~41 s, a 87 % improvement and near‑100 % hit rate.

Node.js

Cache the node_modules directory. Build time reduces from ~58 s to ~29 s, a 50 % gain with almost 100 % hit rate.

Golang

Cache the $GOCACHE directory. Build time drops from ~117 s to ~18 s, an 84.6 % improvement.

GCC (ccache)

Compress the CCACHE_DIR and store it on CFS. Cache hits eliminate recompilation, dramatically speeding up builds.

Custom Image Cache Support

Two generic cache atoms are provided: Restore (pull and unpack cache before the build) and Save (compress and push cache after the build). This works for Maven, Node.js, and other build steps.

Future Plans

Expose cache enablement and key configuration per build atom.

Implement incremental push for different language caches.