Why Grafana Tempo and TraceQL Are Game‑Changers for Lightweight Tracing

Temp Introduction

When it comes to distributed tracing, many are familiar with SkyWalking and Zipkin. This article introduces Grafana Tempo, a lightweight, cost‑effective tracing solution that integrates tightly with Grafana, Prometheus, and Loki, making it suitable for small‑to‑medium teams.

Using Grafana's Redis Data Source plugin to monitor Redis

Grafana 8: unified alerts, real‑time streaming, new visualizations

Streaming sensor data to Grafana with MQTT and Grafana Live

Loki challenges the ELK stack

Loki 2.3 releases with a 15× query speed boost

Mica 2.5.4 adds Prometheus http‑sd support

The author's open‑source Spring Cloud microservice toolkit mica also extends these components.

mica‑prometheus (supports http‑sd discovery via service registry)

mica‑logging (customizable log label format and push to Loki)

Grafana Tempo is an open‑source, highly scalable tracing system that stores and queries traces efficiently. It integrates deeply with Grafana, Prometheus, and Loki, and works with any open‑source tracing protocol such as Jaeger, Zipkin, OpenCensus, and OpenTelemetry. Tempo currently does not support the SkyWalking agent (see issue #466).

Tempo 2.0 adds the TraceQL query language, complementing Loki's LogQL. TraceQL enables complex, data‑type‑aware queries to extract and search traces more effectively.

TraceQL Motivation

Distributed traces contain rich information for error tracking, root‑cause analysis, and performance tuning. While tools can capture data automatically, extracting value from it is challenging. Tempo’s metric generator can aggregate traces into service graphs, and TraceQL allows interactive navigation from high‑level metrics to the underlying traces.

Traces form an event stream with a tree structure (root, branches, leaves) and arbitrary key‑value data with timestamps. Existing tracing solutions struggle to express complex queries across this structure, prompting the design of a new language inspired by PromQL and LogQL.

Example TraceQL query: { .db.statement =~ "INSERT.*"} | avg(duration) > 1s This query finds traces involving database INSERT statements whose average duration exceeds one second—something difficult to express in other systems.

With TraceQL you can craft sophisticated queries to analyze trace data more effectively.

Current Tempo search offers only basic text matching, whereas TraceQL is data‑type aware, providing flexible operations on text, integers, and other types.

Tempo 2.0 will adopt a new storage format based on the open‑source columnar Apache Parquet project, enabling efficient trace storage and broad compatibility with databases, data pipelines, and analytics tools. Combined with TraceQL, this unlocks deeper insights for organizations.