Improving Continuous Integration: Reducing Build Times, Enabling Automated Testing, and Managing Estimations

The previous article "Continuous Integration 9 Questions" highlighted many unresolved issues that have become critical under the new CI constraints.

1. Build time too long: A full C++ project compile takes 40 minutes, far exceeding the CI goal of completing compilation, packaging, and automated test execution within 15 minutes. To address this, a three‑person team built a company‑level C/C++ compile cluster over three weeks. Developers now run localbuild.sh on their machines, which submits the current SVN revision and patch file to a server; the server queues the job and distributes it to idle machines for fast compilation and packaging, providing immediate feedback on errors.

2. Developers cannot run automated tests: Test cases reside in a separate test repository with restricted access. In an iterative development model, developers need to run these tests before committing code. The test team supplied a script that developers can invoke to execute the automated tests, and they prepared one‑click test data generation so a single local shell script sets up the required environment.

3. Automated test strategy: The team agreed on three levels: (1) Unit tests – not enforced initially due to lack of practice; (2) Module/subsystem tests – aim for high automation coverage, with developers and testers jointly deciding which scenarios need tests; (3) System‑level tests – discussed and defined jointly before new feature development.

4. Criteria for a feature to be ready for testing: The developer must have the test team experience the feature on their machine, and all relevant automated tests (including new or modified ones) must pass. Once satisfied, the CI build number is sent to the test team, who download the build package for comprehensive functional testing, and manual packaging is strictly prohibited.

5. Performance and stress testing: To obtain early quality feedback, the team runs a subset of critical cases with predefined metrics and thresholds, generating alerts on anomalies and performing weekly manual comparisons. This balances rapid feedback with manageable cost.

6. Changing original effort estimates: Original estimates should not be altered unless the initial assumptions are invalid. New requirements require quick estimation for planning, while re‑estimating already‑estimated items is discouraged due to the limited added value and the decision‑making overhead it introduces.