How Pupu APM Halved Costs Using Head Sampling

Background

Rapid growth of Pupu’s business led to microservices, cloud‑native, and container adoption. Almost all services report traces to Pupu APM, causing a sharp increase in data volume, higher IT costs, and degraded query performance.

Initial Cost‑Reduction Measures

Storage backend switched from Elasticsearch to ClickHouse, reducing storage cost by two‑thirds with comparable performance (see “Pupu APM ClickHouse Storage Practice”). A T+1 tail‑sampling step further lowered storage cost and improved query speed (see “Storage Cost Reduced 80 %, Query Efficiency Up 5×, Pupu APM Link Sampling Practice”).

Motivation for Head Sampling

Even after these optimizations, costs remained high. Discarding meaningless spans at the source (head sampling) can reduce load on the message queue, OAP, and ClickHouse, improve signal‑to‑noise ratio, and boost query performance.

Challenges of Traditional Head Sampling

Conventional head sampling hashes the TraceId and applies a fixed ratio per service. This simple logic cannot guarantee complete traces. Figure 1 shows that with services A, B, C sampled at 50 %, 25 % and 100 % respectively, only one of four possible traces remains intact.

Users also require:

Full retention of error traces

Full retention of slow database or interface spans

Full retention of core business flows (search, add‑to‑cart, order, payment)

Ability to ignore non‑core spans

Traditional head sampling cannot satisfy these requirements, so a custom solution was designed with the goal of halving total cost while keeping the APM experience for 99 % of services unaffected.

Data‑Driven Insights

Analysis of massive span data revealed:

80 % of spans originate from 20 % of services (Pareto principle).

Consumer services that asynchronously consume binlog generate roughly half of all spans.

Two daily peaks in span submission occur around 11 am and 5 pm, matching end‑user traffic.

Non‑core services (e.g., certain big‑data services) produce excessive spans.

User‑interest spans constitute less than 5 % of the global span volume.

These observations guided the sampling strategy: focus head sampling on the top 20 % of services, especially consumer services; shave the two peak periods; apply higher sampling rates to non‑core services; and drop spans that users rarely care about.

Sampling Rule Design

Any trace containing an error. Spans exceeding latency thresholds (e.g., XXL‑Job > 3 s, Kafka consumer > 2 s, HTTP > 500 ms, Dubbo > 250 ms, Elasticsearch > 250 ms, MySQL > 200 ms, Redis > 200 ms). Core business call chains (search, add‑to‑cart, order, payment).

Figure 2 shows the UI where users can set per‑service rules; error spans are forced to be retained by default.

Sampling Strategy Design

Four strategies are supported (Figure 3):

Default : respects upstream sampling flag; if none, applies full‑report or ignore‑report rules; always retains errors.

Span Full Sampling : retains all spans of the service.

Span Fixed Sampling : retains a fixed percentage of spans (e.g., 80 %). Suitable for non‑core services that still need some visibility.

Peak‑Time Sampling : applies fixed‑rate sampling only during peak windows (10:00‑12:00 and 16:30‑18:30). Ideal for core services that require both meaningful span analysis and daily business analysis.

Sampling Coloring Design

Sampling is performed only at head services. Once a head service is sampled, all downstream calls inherit the sampling mark, ensuring trace completeness. The rule guarantees that the sampled span and all subsequent calls are retained. Figure 7 illustrates three scenarios: strategy hit (forced retain), strategy miss (forced ignore), and default (no explicit rule).

Implementation Example

public class OwlSamplingCarrierItem extends CarrierItem {
    public static final String HEADER_NAME = "Owl-sampling";
    private ContextCarrier carrier;
    public OwlSamplingCarrierItem(ContextCarrier carrier, CarrierItem next) {
        super(HEADER_NAME, carrier.getSampling() + "", next);
        this.carrier = carrier;
    }
    @Override
    public void setHeadValue(String headValue) {
        // If a sampling strategy is set for the current service, ignore upstream flag
        if (OwlHeadSampleConfigProperties.trySamplingSwitchIsNull()) {
            carrier.setSampling(Integer.valueOf(headValue == null ? "1" : headValue));
        }
        // Errors have highest priority; force retention
        if ("999".equals(headValue)) {
            carrier.setSampling(999);
        }
    }
}

Results

After deploying head sampling, daily span reports dropped from several hundred billion to 500 billion. Table 1 shows that the number of machines for ClickHouse, Kafka, and the analysis cluster each fell by half, achieving roughly a 50 % reduction in total cost while keeping the APM experience for most services unchanged.

Conclusion and Outlook

The head‑sampling practice demonstrated that a well‑designed sampling layer can dramatically cut infrastructure costs and improve query performance. Internal statistics show that user‑interest spans are under 5 % of total volume. Future work will explore increasing the discard ratio further without compromising users’ ability to troubleshoot issues.