Backend Build Compilation Optimization for QQ Music: Reducing Build Time from 40 Minutes to 30 Seconds

Background: As the QQ Music backend functionality grew, the codebase expanded and compilation time increased dramatically, taking 30‑40 minutes for a full build, severely affecting development and release efficiency.

Solution Idea: After evaluating KM and other online solutions, the team chose a minimal‑impact approach using distcc+ccache . To reduce disk usage, both the source code and the ccache directory were placed on a tmpfs filesystem.

Implementation Process:

1. Install ccache on five development machines:

#!/bin/sh tar -xvf ccache-3.2.4.tar.gz cd ccache-3.2.4 ./configure && make && make install cd /usr/local/bin ln -s ccache /usr/local/bin/gcc ln -s ccache /usr/local/bin/g++

2. Install distcc (Python required first):

#!/bin/sh [ -d distcc-distcc-3.1 ] || unzip distcc-distcc-3.1.zip cd distcc-distcc-3.1/ ./autogen.sh ./configure&&make&&make install # start daemon distccd --daemon -a 192.168.1.1 -a 192.168.1.2 -a 192.168.1.3 -a 192.168.1.4 -a 192.168.1.5

3. On the main compilation host (e.g., 192.168.1.1) mount tmpfs and configure environment variables:

# mount tmpfs mkdir -p /data/home/user_00/AutoBuild mount tmpfs /data/home/user_00/AutoBuild -t tmpfs -o size=10g # set environment export DISTCC_HOSTS="localhost/16 --localslots=12 192.168.1.2/5 192.168.1.3/5 192.168.1.4/5 192.168.1.5/5" export CCACHE_PREFIX=distcc export MAKEFLAGS=\"-j28\" # set ccache cache directory echo "cache_dir = /data/home/user_00/AutoBuild/.ccache" >> /data/home/user_00/.ccache/ccache.conf

Encountered Problems and Fixes:

1. Parallel build errors due to missing dependencies in the Makefile (e.g., cgi must depend on comm ). The solution was to explicitly declare the dependency order.

2. -jN forced in submake: disabling jobserver mode – caused by using a custom QMAKE variable that added its own -j . Removing QMAKE and invoking $(MAKE) directly solved it.

3. warning: jobserver unavailable: using -j1. Add '+' to parent make rule. – also due to invoking make -f makefile directly instead of $(MAKE) . Switching to $(MAKE) propagated the jobserver flags correctly.

4. ccache -s reported many “unsupported compiler option” messages because the project used -MM to generate dependency files, which ccache does not understand. The workaround was to avoid -MM and perform a full make clean before rebuilding.

5. Cache misses caused by the __TIME__ macro. Setting CCACHE_LOGFILE revealed the issue; the fix was either to remove time‑related macros or add sloppiness = time_macros to ccache.conf .

6. --localslots not taking effect, leaving only four local ld processes. The team patched lock.c in the distcc source to increase the default.

7. Inconsistent behavior of make clean all versus make clean; make all . The problem stemmed from parallel execution of the two targets. Three remedies were tried: using the sequential form, adding .NOTPARALLEL: (which disabled all parallelism), and defining a new target clean_all that splits the steps.

8. The CIS build system forced -j4 and ignored changes. Overriding the default required setting MAKEFLAGS manually.

9. Using distcc ccache g++ directly performed worse than setting CCACHE_PREFIX=distcc because the former forced remote caching for all objects, bypassing the local cache benefits.

Optimization Results:

• Without ccache, the first build took about 2 minutes 30 seconds.

• With ccache enabled, the build time dropped to roughly 30 seconds, surpassing the original goal of completing the build within one minute.

Future Optimizations:

1. Clean up duplicate files across the project.

2. Remove unnecessary #include statements, as excessive includes increase precompiled header size and slow compilation.

3. Transition from 32‑bit to 64‑bit builds.

References:

• “Backend Build Tools: ccache and distcc”

• Linux articles on using distcc and ccache to accelerate project compilation

• “Memory Filesystem Usage and Examples: ramdisk, ramfs, tmpfs”

• http://code.google.com/p/distcc/

• https://ccache.samba.org/

• http://stackoverflow.com/questions/8496135/parallel-makefile-require-depency-ordering