Why CTOs Must Pay Attention to Unstable Automated Tests: Lessons from Aviation and Google

Human nature tends to ignore alarms, but CTOs must stay vigilant about unstable automated test failures, just as pilots should heed cockpit warnings.

Recent aviation reports show Boeing 737 pilots frequently disregarding pressure‑loss alerts, leading to fatal accidents such as Helios Airways Flight 522. The same complacency can occur in software development when flaky tests are treated as false positives.

Analysis from Travel Weekly and NASA's ASRS indicates that many 737 pilots ignore critical cabin‑pressure alarms, mistaking them for pre‑flight warnings.

Since the 2017 DevOps wave, automated testing has become a top priority for CTOs, with companies like Microsoft historically investing heavily in test development and maintaining near‑equal ratios of developers to test engineers.

Even industry leaders such as Google experience flaky tests: about 1.5% of their test suite is unstable, and roughly 16% of tests show some instability, causing significant friction in the development workflow.

Google combats flakiness through strict engineering discipline, pre‑ and post‑submit testing, and five mitigation tactics: rerunning failed tests, automatic retries, marking tests after three consecutive failures, isolating problematic tests, and auto‑creating bugs for unstable tests.

Additional strategies include monitoring tools that flag high‑instability tests, automatically isolating them from critical paths, and dedicated teams that provide timely information about test reliability.

The cost of flaky tests includes delayed code submissions, extra investigation effort, and potential release of buggy code when false alarms are ignored.

Ultimately, reducing flaky tests restores confidence in the test suite, improves developer productivity, and ensures that automated testing continues to deliver its intended quality guarantees.