CNB Volumes Cache: Accelerating Large-Scale Compilation with Copy-on-Write

This article explains how CNB (Cloud Native Build) uses Docker volumes with Copy-on-Write (CoW) mechanism to dramatically accelerate compilation of large codebases like AOSP (125GB, 14,000+ repositories).

The team faced challenges with long clone times, high storage usage, low efficiency, and limited concurrent builds. After solving the clone speed issue with git-clone-yyds plugin, they still needed to address the 46-minute full compilation time.

CNB implements cache through Docker volumes mounted to build containers. The system supports multiple caching strategies: read-write, read-only, copy-on-write, copy-on-write-read-only, and data (temporary volume). The Copy-on-Write approach is particularly powerful for concurrent builds.

Configuration is simple - just modify the docker.volumes field in the pipeline configuration:

build_config: &aosp_build_config
  runner:
    cpus: 64
  docker:
    build: .ide/Dockerfile
    # Mount volume cache as copy-on-write
    volumes:
      - out:copy-on-write
  stages:
    - name: build
      env:
        BUILD_HOSTNAME: cnb-build
        shell: |
          source build/envsetup.sh
          lunch aosp_arm-eng
          make -j64
      script: bash -e -c "${shell}"

With 6 concurrent pipelines using Copy-on-Write caching, the team achieved 100% incremental compilation, reducing build time from 46 minutes 40 seconds to just 1 minute 30 seconds.

The technical implementation uses overlay filesystem: CNB creates cache folders in /data/cache on the host machine, and for Copy-on-Write mode, it establishes upper dir, work dir, and merged dir in /data/copy-on-write. These are mounted via "mount -t overlay" and then attached to containers via Docker -v parameter.

When multiple pipelines use the cache concurrently, CNB creates multiple CoW folder sets (with shared lower dir), allowing parallel builds to use the cache simultaneously without conflicts.

Beyond caching, CNB also offers remote development capabilities combining these features: second-level workspace startup via git-clone-yyds, fast incremental compilation using volumes cache, and support for multiple developers working concurrently on the same project with shared caches.