15 Years of DevOps Evolution: From CruiseControl to AI‑Driven CI/CD

Since 2008 the author has participated in multiple DevOps system projects, reflecting on their origins, design choices, and the problems they solved, while re‑examining those issues from today’s perspective.

Early CI Experience: CruiseControl and Hudson

Working at a foreign telecom equipment company, the author first encountered continuous integration through CruiseControl, learning about jobs, triggers, artifacts, and daily builds. CruiseControl felt cumbersome, so the team switched to Hudson, which simplified configuration.

Both tools are Java‑based, run from a WAR file, support plugins, and provide job and build concepts with scheduling and authentication.

The primary use cases were building large embedded software packages and running automated tests.

Building Large Embedded Packages

Embedded telecom software, often tens of millions of lines stored in a remote SVN, required lengthy builds on physical servers before Docker and fast VMs existed. Build scripts used chroot to select toolchain versions, and artifacts were stored via CI. Distributed compilation with distcc improved speed. Modern equivalents use Bazel and containers, but challenges like caching and scheduling remain.

Automated Testing with RobotFramework

Automated tests dominated CI usage. The team adopted RobotFramework, storing test cases and tools in SVN, fetching them during builds. This practice persists today, now using Git and container clusters.

Jenkins and Custom Plugins

Open‑source Jenkins replaced Hudson. As job count grew, the team needed custom plugins for device locking, pool management, and rapid configuration updates across thousands of jobs. They reduced plugins to a minimal set, built custom Jenkins plugins, and created a unified module CI tool, cutting maintenance effort to a single person for hundreds of developers.

Emerging Problems with Jenkins

Jenkins’ single‑master architecture caused downtime during updates, and increasing job frequency (multiple daily releases) made plugin updates disruptive. Scaling issues included master disk I/O bottlenecks and the difficulty of extending Jenkins UI for team‑level views.

Custom Platform + Jenkins + HDFS

In 2011 the team designed a new platform to address Jenkins master high‑availability, unify processes, improve test device utilization, and provide data for analytics. Jenkins became a hidden execution engine behind a multi‑master architecture, with jobs created and destroyed on demand, and artifacts stored in HDFS.

Metrics, Monitoring, and AI

Early dashboards visualized build status, commit frequency, success rate, and mean time to repair, resembling today’s DORA metrics. The team built a code‑review portal similar to Sonar, aggregating complexity, duplication, and static analysis results. Around 2016‑2017 they integrated ELK and AI techniques for real‑time log analysis and performance improvement.

Startup CI/CD Practices

In 2017 a 10‑person startup adopted GitLab (mirrored from GitHub) for source control, Jenkins for CI (supplemented by GitLab‑CI for tag triggers), and a simple artifact repository for RPM, APK, and PyPI packages. They enforced fully automated, event‑driven pipelines, stored all configurations as code, and used a trunk‑based branching model with short‑lived feature and release branches.

From Development to Operations

When moving to SaaS delivery in 2017, the team separated offline (development) and online (production) environments, introduced DNS‑based configuration, and built a multi‑master Jenkins deployment to achieve zero‑downtime upgrades. They added monitoring, gray‑release, and A/B testing capabilities, recognizing that DevOps requires tight coupling between development artifacts and operational resources.

Private‑Cloud Delivery and BizDevOps

From 2018 onward the author worked on a private‑cloud security product, confronting version‑based delivery and multi‑tenant requirements. They defined a domain model centered on systems and versions rather than code branches, built custom delivery portals, and later contributed to Alibaba Cloud’s “BizDevOps” whitepaper, proposing a foundational model for DevOps platforms.

Future Outlook: AI and Platform Engineering

The author envisions AI‑assisted test generation, automated performance analysis, and intelligent security advisors, while emphasizing that high‑value tools and long‑term data remain essential assets for developers.

Conclusion

Regardless of emerging technologies, the core goals of DevOps remain: faster value delivery for teams and reduced cognitive load for developers.