Sampling at Ten‑Million QPS: From Full to Intelligent Adaptive Sampling

Why Not Keep Everything

Full (100%) sampling records every span of every request, which works for low‑traffic or testing environments, but at 10 million QPS with an average of 30 spans per request and ~500 bytes per span, the raw data rate reaches ~150 GB/s, or about 13 PB per day, overwhelming storage, performance, pipeline, signal‑to‑noise, and query costs.

First wall – storage explosion: PB‑scale data incurs massive disk, replica, backup, and tiering costs, yet 99% of it is never queried.

Second wall – performance overhead: instrumentation consumes CPU, memory, and bandwidth; a few percent throughput loss at scale translates to many machines.

Third wall – collection pipeline pressure: collectors, Kafka buffers, and back‑ends must all scale to handle 150 GB/s.

Fourth wall – signal‑to‑noise ratio: 99% of requests are identical fast successes, offering little diagnostic value.

Fifth wall – query and analysis cost: larger datasets mean larger indexes, slower aggregations, and more complex tiering.

First Knife: Dice at the Entry

Head‑based (or entry‑point) sampling decides at request arrival whether to keep the trace, typically using a fixed probability such as 1%. The decision is stored in the trace context (e.g., the sampled flag in W3C traceparent or Zipkin’s B3 header) and propagated downstream so the whole trace is either kept or dropped.

Consistent sampling improves stability by hashing the trace ID and sampling based on the hash result, ensuring the same decision even if the flag is lost. OpenTelemetry implements this with the TraceIdRatioBased sampler, often combined with ParentBased to respect upstream decisions.

Advantages: cheap, predictable, zero buffering, and easy budgeting (1% reduces storage and bandwidth to 1%).

Blind spot: the decision is made before the request outcome is known. For a failure rate of 0.1% and a 1% sampling rate, the chance of capturing a failing trace is 0.001% (roughly one in 100 000 failures). Bias: a high‑traffic endpoint gets many samples while a low‑traffic critical endpoint may receive none.

Adding a Fuse at the Backend

Rate‑limiting sampling caps the number of samples per service per second, protecting collectors during traffic spikes. A typical pattern is “probability + limit”: low‑traffic requests are sampled proportionally, while high‑traffic bursts hit a hard ceiling. AWS X‑Ray exemplifies this with a per‑rule reservoir (minimum per‑second samples) plus a percentage‑based overflow.

Wait for the Story to Finish

Tail‑based sampling defers the decision until the trace is complete, allowing policies such as:

Keep all error traces.

Keep all traces whose latency exceeds a threshold (e.g., P99).

Keep traces that hit rare branches or specific tags.

Randomly sample the remaining normal traces.

This approach requires buffering the entire trace in memory and routing all spans of the same trace to the same collector instance, which introduces three major costs:

Memory pressure: collectors must hold all in‑flight traces within a sliding window.

Routing complexity: a consistent‑hash load balancer (e.g., OpenTelemetry Collector’s load‑balancing exporter) ensures all spans of a trace land on the same node.

Decision latency and operational complexity: decisions are delayed by seconds to minutes, and the overall architecture is an order of magnitude more complex than head‑based sampling.

The OpenTelemetry tail_sampling processor chains multiple policies (status_code, latency, probabilistic, rate_limiting, string_attribute) with an OR semantics, enabling retention of errors and slow traces while still sampling normal traffic.

Let Sampling Learn to Choose

Intelligent (adaptive) sampling adds automation to the knob:

Target‑rate adaptive: each operation defines a desired sample count (e.g., at least 10 traces per second). The system observes traffic and dynamically adjusts per‑operation rates, as implemented by Jaeger’s remote adaptive sampler.

Priority‑bias sampling: traces flagged as errors, high latency, VIP tenant, or new feature branches receive higher rates, up to 100%.

Dynamic remote configuration: sampling policies are stored in a central config service and pushed to SDKs without service restarts (OpenTelemetry remote sampler, Jaeger remote sampler).

Forced sampling / debug: a special header or trace‑parent flag can force full sampling for a specific request, useful for targeted debugging.

Head‑tail combination: cheap head‑based coarse filtering (e.g., health checks) reduces load, while tail‑based fine filtering preserves valuable error/slow traces.

ML‑driven bias and exemplars: anomaly detection can automatically boost sampling for unusual traces, and exemplars link metric outliers to the corresponding trace.

Standing on Giants' Shoulders

The concept dates back to Google’s 2010 Dapper paper, which used a 1/1000 sampling rate at massive scale and showed that statistical sampling remains sufficient for latency analysis and path reconstruction.

Modern ecosystems provide concrete implementations:

OpenTelemetry: head samplers ( TraceIdRatioBased, ParentBased) and collector‑side tail_sampling, load‑balancing exporter.

Jaeger: remote adaptive sampling with per‑operation target rates.

Zipkin: B3 header propagation for head‑based sampling.

AWS X‑Ray: reservoir + rate combination.

Grafana Tempo / Honeycomb Refinery / Lightstep: productized dynamic tail sampling.

All share the same underlying goal: balance cost and fidelity for the given scale.

Turn the Knob to the Right Position

Starting from full sampling (complete but unsustainable), moving to head‑based probability + rate‑limiting (cheap but blind), then to tail‑based sampling (accurate but costly), and finally to intelligent adaptive sampling (cost‑effective, accurate, and self‑adjusting), the core proposition remains: sampling is the fundamental cost‑fidelity knob that should keep the “important” traces (errors, slow tails, rare paths) while discarding the boring majority.