CI/CD Observability via OpenTelemetry at Grafana Labs

In this article the author introduces the concept and importance of CI/CD observability. By using observability, teams can proactively resolve issues, make smarter decisions, and increase confidence in software releases. The piece also mentions common CI/CD problems such as instability, performance regressions, and misconfigurations.

The author presents GraCIe , a Grafana‑based application plugin that provides an easy‑to‑understand view of CI/CD systems. GraCIe leverages Grafana Tempo, Grafana Loki, and Prometheus via OpenTelemetry to integrate seamlessly with almost any CI/CD platform, delivering unprecedented insight.

Why You Should Care About CI/CD Observability

CI/CD observability, a subset of overall observability, focuses on the software development lifecycle and helps ensure processes are reliable, relevant, and understandable. Benefits include:

Proactive problem solving: Detect and fix issues before they escalate, saving time and resources.

Better decision making: Detailed pipeline data informs resource allocation, process changes, and tool adoption.

Increased confidence: Clear insight into pipelines reduces deployment anxiety and fosters a culture of continuous improvement.

Accountability and transparency: Every step is traceable, enabling root‑cause analysis rather than symptom treatment.

Common Issues

Three frequent challenges that disrupt smooth CI/CD operation are instability, performance regression, and misconfigurations.

Instability (Flakiness)

Flaky tests produce inconsistent results without code changes, often due to external dependencies, environment problems, or nondeterministic test conditions.

Performance Regression

As pipelines grow more complex, performance can degrade because of inefficient test execution, redundant operations, or code and test bloat that increase build times.

Misconfigurations

Even well‑designed pipelines can fail due to sub‑optimal test ordering or insufficient resource planning, leading to bottlenecks and reduced throughput.

"Sub‑optimal" refers to situations, decisions, or outcomes that are not ideal, indicating room for improvement.

The Importance of DORA Metrics

DORA metrics—Deployment Frequency, Mean Lead Time, Mean Time to Recovery, and Change Failure Rate—are the industry standard for measuring software delivery effectiveness and health.

How We Started Optimizing CI/CD Observability

Grafana Labs began by focusing on the grafana/grafana repository, encountering flaky tests in both Grafana OSS and Enterprise, and dealing with a Drone CI tool that often stalled. To fill the gap, a custom Prometheus exporter was created, feeding new data into dashboards that quickly surface pipeline health.

Two example changes made to ensure observability became part of the CI/CD process:

Alert when a protected branch build fails, ensuring the repository can always be built.

Track restarts not triggered by code changes to detect underlying instability.

Scaling Our Observability Work

Success with the initial repository attracted interest from other Grafana Labs teams. The goal is to extend observability to many repositories without adding operational overhead, enabling seamless integration for all teams.

Building with OpenTelemetry

A custom OpenTelemetry receiver was developed for the Drone CI tool, laying the groundwork for broader CI/CD observability solutions and anticipating a future universal standard for telemetry data access across CI/CD systems.

Enhancing CI/CD Observability in Grafana

The result is GraCIe , a Grafana application plugin that simplifies evaluation of build performance, identifies inconsistencies in test results, and analyzes build output. By leveraging Grafana Tempo, Loki, and Prometheus, and relying on OpenTelemetry, GraCIe can work with virtually any CI/CD platform without custom configuration.

The Future Is Interoperable

Grafana Labs is just beginning with GraCIe, aiming not only to solve current challenges but also to shape the future of CI/CD observability, envisioning a world where every Grafana user can effortlessly obtain the tools and insights they need regardless of the underlying CI/CD platform.

For more details, see the OpenTelemetry proposal (https://github.com/open-telemetry/oteps/pull/223) and share feedback via the provided form.