Accelerating C++ Build Times with distcc, ccache, and Docker Containers

In 2011, while working at Baidu, the author noticed that a large C++ project required 40 minutes per build, which hindered rapid feedback in continuous integration. To address this, a first‑generation distributed compilation tool based on distcc and ccache was created.

The article demonstrates how to use a distcc server container to distribute compilation load across heterogeneous nodes (laptops, desktops, and a Mac) and how ccache provides local compilation caching.

Performance before acceleration : Building LLDB (git tag llvmorg‑7.1.0) on a Lenovo ThinkPad T460s with Fedora 29, GCC 8.2.1, and clang 7.0.1 took about 72 minutes (real time) using make -j 4 .

Performance after acceleration : Adding ccache 3.4.2 and distcc 3.2rc1, and using Ninja with 10 parallel jobs, reduced the build time to roughly 22 minutes, a 3.22× speed‑up in real time, 4.56× less CPU time in user mode, and 2.7× less in kernel mode.

Tip 1 – Distribute compilation load : distcc can dispatch compile jobs to other machines, but requires identical compiler installations on client and server; containers simplify this requirement.

Tip 2 – Use a distcc server container :

ENTRYPOINT [
  "distccd",
  "--daemon",
  "--no-detach",
  "--user", "distcc",
  "--port", "3632",
  "--stats",
  "--stats-port", "3633",
  "--log-stderr",
  "--listen", "0.0.0.0"
]

Default server parameters can be overridden, e.g.:

# Allow any client, set nice level and job limit
CMD [
  "--allow", "0.0.0.0/0",
  "--nice", "5",
  "--jobs", "5"
]

Run the image without building it yourself:

docker pull konradkleine/distcc:fedora29

Start the container:

docker run \
  -p 3632:3632 \
  -p 3633:3633 \
  -d \
  konradkleine/distcc:fedora29

Compile LLDB using the distributed setup:

git clone https://github.com/llvm/llvm-project.git ~/dev/llvm-project
export LOCAL_DISTCC_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' localdistcc)
mkdir -p ~/dev/llvm-builds/release-gcc-distcc
cd ~/dev/llvm-builds/release-gcc-distcc
cmake ~/dev/llvm-project/llvm \
  -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_USE_LINKER=gold \
  -DLLVM_ENABLE_PROJECTS="lldb;clang;lld" \
  -DCMAKE_C_COMPILER=/usr/bin/gcc \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
  -DCMAKE_C_COMPILER_LAUNCHER="ccache;distcc" \
  -DCMAKE_CXX_COMPILER_LAUNCHER="ccache;distcc"
  ninja lldb -j $(distcc -j)

Adjust DISTCC_HOSTS to control job distribution, e.g. export DISTCC_HOSTS="fasthost/8 slowhost/2 localhost" . Verify ccache activity with ccache --show-stats .

FAQ highlights :

Check that all hosts are serving distcc (e.g., nc -zv $i 3632 or nmap -A $i/32 -p 3632 -Pn ).

Run Docker with --allow <YOUR_HOST> for tighter security.

Both distcc and ccache can be used together or separately by adjusting CMake launchers.

Alternative tools such as Icecream provide similar distributed compilation without requiring identical toolchains on each node.