Design and Optimization of a Distributed Build System for the MMO Game "Nishuihan"

The article originates from a presentation at the MTSC conference by the LeiHuo Test Center, using the Nishuihan PC game as a case study to illustrate a distributed build system tailored for large‑scale game projects.

Project Background : The original build workflow spent roughly 90% of its time on art asset processing due to the project's 400 GB of raw assets and diverse resource types, causing long build times and low testing efficiency for a development team of about 200 people.

Art Resource Handling : The team categorized over 70 000 assets into four main processing rules, eliminated unnecessary dependencies (e.g., between .mat and .prefab files), and introduced incremental packaging by syncing only modified assets to a trunk directory, enabling incremental builds.

Incremental Packaging : A nightly process records modified assets between builds, uploads processed results to the trunk, and the client build consumes these ready‑to‑use resources, dramatically cutting redundant work.

Distributed Build : To parallelize the heavy resource processing stage, three scheduling strategies were defined: (1) dispatching independent tasks to any idle machine, (2) enforcing intra‑machine task dependencies (e.g., compilation after SVN update), and (3) enforcing inter‑machine dependencies (e.g., packaging after processing). Buildbot, a Python‑based open‑source scheduler, was customized to implement these strategies.

Optimization Steps : The team identified bottlenecks such as long waits for tool compilation, uneven resource distribution, and sequential shader packaging. Solutions included splitting tool compilation into separate executables, partitioning large asset groups (e.g., splitting the character directory into 28 sub‑tasks), parallelizing shader compilation, and overlapping WDF upload with packaging.

Multi‑Team Support : A lightweight "slimtrunk" mechanism was created to provide functional teams (design, art, QA) with a fast‑sync local client by checking out a dedicated SVN folder containing only the necessary resources, code, and executables.

Experience & Lessons : The authors emphasize designing requirements with the distributed pipeline in mind, avoiding hard‑coded dependencies, and separating long‑running checks from the main build to prevent unnecessary delays.

Outlook : A new build framework from LeiHuo Build Platform will replace Buildbot, aiming to further streamline distributed scheduling and provide broader support for future projects.