How to Cross the DevOps Gap: Causes, Key Metrics, and Organizational Strategies

In a previous article "The Smoke Pipe Curve in DevOps Implementation" we discussed how few enterprises can quickly cross the "gap" of the smoke pipe curve and achieve continuous improvement. This article asks how those successful companies actually bridge that gap.

1. Why does the gap appear?

When an organization declares "We are adopting DevOps", it often immediately purchases tools and hires DevOps engineers to automate existing processes (e.g., automatic test submission, notification emails). Initial automation yields benefits, and once the process is optimized to a certain point, attention shifts to "test acceleration" and automated testing begins.

The first problem encountered in automated testing is the automation of test environment preparation. The benefit gained depends on how painful the manual environment setup was originally.

After the environment is ready, testers start writing many automated test cases. The first two months look promising, but soon the curve reaches its first inflection point, and adding more test cases no longer yields noticeable improvements. At this stage, organizations either continue the same approach (risking diminishing returns) or remain stuck on the plateau.

At this plateau, the DevOps strategy becomes "pick the low-hanging fruit": (1) avoid large, disruptive changes; (2) start with easy wins.

Why does the curve initially rise? Process automation reduces transactional costs in each delivery iteration, freeing time for value‑adding activities. However, after shortening the iteration cycle to a certain limit, the proportion of value‑adding versus transactional activities tends to revert, causing frustration.

Continuous delivery requires many automated test cases, often end‑to‑end tests. As the number of such tests grows, maintenance costs explode, leading to a platform‑stage where the cost‑benefit ratio is questioned.

End‑to‑end automated tests are costly to run, maintain, and diagnose, and many test engineers lack the skills to write lower‑level tests. Even when they can, collaboration and cultural factors often prevent timely information flow, hindering the quality of low‑level tests. This is the core of the gap.

2. Two key considerations for implementing DevOps

First: Properly understand "metrics".

Metrics have three attributes: observability, actionability, and timeliness (lead vs. lag). Some metrics are directly observable but not actionable (e.g., defects per KLOC). Others are lead indicators that can guide behavior, such as code review count or coding standards compliance.

Another example: defects found between test submission and production release are lagging; the number of self‑test cases can be a leading, actionable metric, though causality is not guaranteed.

Measuring software development efficiency is notoriously hard; no universal metric exists. The usefulness of a metric depends on organizational capability, product type, and skill levels. Metrics closer to the right side of the diagram are more guiding, while those on the left are outcome‑oriented.

When designing improvement plans, one must consider which set of metrics to choose, how changes in one metric affect others, and the delayed effect of lead indicators.

Collecting metric data incurs cost; early on, sampling can provide credible data with lower expense, though it introduces bias and risk.

Second: Seek structural changes in the system itself ("second‑order change"). Changing who uses automated test cases—from testers to developers—requires test cases to be fast, easy to invoke, trustworthy, and timely. This shift often entails moving tests from high‑level end‑to‑end down to integration, component, or unit tests, altering roles and responsibilities.

Such structural changes can improve the state of automated testing and lead to a positive feedback triangle.

3. Where is DevOps heading?

DevOps promotes a common language that enables tighter collaboration across roles, aligning teams around business goals rather than functional silos. Organizations should structure around business units, breaking both "functional walls" and "business unit walls" when possible, and ensure alignment among many small units—similar to micro‑service architecture.

Effective alignment can be inspired by the "Amoeba Management" model, where small, autonomous units operate with clear profit‑and‑loss responsibility, fostering self‑organization.

To achieve such transformation, three aspects must be addressed: software architecture, organizational mechanisms, and collaboration infrastructure. Detailed case studies are available in "Continuous Delivery 2.0".

4. Summary

Optimizations that do not change system structure are relatively easy, but once they reach their limits, a second‑order change is required to achieve a breakthrough and reach the next level of performance.