Scaling Link Tracing Storage: From Centralized Elasticsearch to Tiered Architecture

During a pre‑sale load test, the tracing system collapsed before the business services because all spans were written directly to a single Elasticsearch (ES) cluster. Under normal traffic the cluster handled hundreds of millions of spans per day, but a 20× traffic spike caused bulk queues to overflow, disk I/O saturation, and node failures, leaving the SRE team unable to query traces.

The author identifies four intrinsic characteristics of tracing data: (1) each request expands into dozens or hundreds of spans, inflating data volume; (2) write‑heavy and read‑light, with less than 0.1% of traces ever queried; (3) trace_id has ultra‑high cardinality, making inverted indexes bloated; and (4) data value decays sharply after a few minutes or hours. These traits make a storage system optimized for flexible full‑text search (like ES) a poor fit for massive write workloads.

Four “walls” arise when scaling ES: write pressure (indexing threads overwhelmed), storage cost (indexes can exceed raw data size), query latency (high‑cardinality indexes force disk reads), and short retention periods (cost constraints force retention of only a few days). The solution must reduce write volume at the source and align storage cost with data value.

First step – sampling. Three sampling strategies are described: head sampling (hash‑based early decision, cheap but may drop error traces), tail sampling (decide after the trace completes, higher value but requires buffering and more memory), and dynamic sampling (adjust rates based on real‑time traffic and service importance). In practice, teams combine head sampling for bulk reduction and tail sampling to guarantee error‑related traces, accepting that aggressive sampling creates observability blind spots.

Second step – hot‑warm‑cold tiered storage. After sampling, remaining spans are stored according to freshness: hot data (last hours to a day) on fast SSD‑backed ES or ClickHouse for sub‑second queries; warm data (few days to two weeks) on lower‑cost nodes or compressed columnar stores; cold data (weeks to months) in cheap object storage (S3/OSS) with minimal query expectations. Automatic tier migration is handled via TTL policies and periodic archiving jobs.

Third step – separating indexes from span payloads. Since most queries retrieve spans by trace_id, a lightweight index mapping trace_id to object‑store blocks suffices. Systems like Grafana Tempo implement this by storing compressed span blocks in object storage and maintaining only a trace_id‑to‑block index, dramatically cutting storage cost while preserving trace_id lookups. Columnar stores such as ClickHouse further improve compression for repetitive fields.

The article then compares four major back‑ends:

ES family (Jaeger/Zipkin/SkyWalking default): flexible search, high cost, suitable for medium scale with generous budgets.

Cassandra family (Jaeger classic): excellent write throughput, fast trace_id lookups, limited secondary indexing.

ClickHouse family: high compression, fast aggregations, lower cost, but less suited for point lookups and higher operational complexity.

Tempo family (object storage + lightweight index): lowest cost, simple scaling, but limited tag‑based search.

In production, large teams often combine these: use object storage with lightweight indexes for cheap long‑term span storage, place recent hot data in ES or ClickHouse for fast debugging, and employ a collector gateway that buffers, batches, and applies sampling to smooth write spikes.

Finally, the author stresses that buffering the write path (e.g., Collector → Kafka → backend) is essential to prevent instantaneous traffic spikes from overwhelming storage, and that both peak‑shaving on the ingestion side and tiered storage on the persistence side are complementary safeguards.

The overarching philosophy is to recognize the unequal value of tracing data, invest storage resources only in the small fraction that truly matters, and balance cost, write performance, and query completeness based on the system’s scale and budget.