Push vs Pull for Million‑QPS Monitoring: Hybrid Edge Pull and Central Push

Why the direction of metric collection matters

When we switched three thousand services to a self‑built StatsD cluster, the collector crashed under a flood of UDP packets, and we even missed the alarm that the monitoring itself was down. The root cause was that three thousand instances pushed metrics to a central collector without any flow‑control, exhausting CPU and memory.

The core question is who initiates the data transfer: the monitored target (push) or the monitoring system (pull). This decision sets the scalability ceiling of the whole monitoring stack.

Push model – the early era

Early monitoring tools such as StatsD, Graphite, collectd, Datadog Agent and InfluxDB + Telegraf all use push. With StatsD, applications emit a line of code that sends metrics via UDP to a local or central daemon, fire‑and‑forget, incurring almost zero cost on the client side.

Push advantages:

Strong penetration : clients initiate outbound connections, easily traversing NAT and firewalls.

Fits short‑lived tasks : a batch job can push its results before exiting.

Real‑time : metrics are sent immediately, no scrape interval delay.

No service discovery needed : the collector passively receives whatever clients send.

Minimal client configuration : only a central address is required.

However, push has structural drawbacks:

The server is passive and cannot control when or how fast data arrives; a fan‑in of many clients can overwhelm it.

It is hard to tell if a missing metric means the target is down, idle, or packets were lost.

UDP provides no delivery guarantee; lost packets are silent.

Global sampling rates are hard to adjust because each client must be reconfigured and restarted.

These issues are tolerable at small scale but become fatal at million‑QPS, ten‑thousand‑instance levels.

Pull model – the cloud‑native default

Prometheus popularized the pull model: each target exposes an HTTP /metrics endpoint, and the Prometheus server scrapes it at a fixed interval (e.g., 15 s).

Pull advantages:

Server controls scrape frequency, providing built‑in flow‑control; the server never gets flooded by clients.

Health detection becomes trivial: a successful scrape yields up=1, a failure yields up=0, turning health into a first‑class metric.

Configuration is centralized; the list of targets and scrape intervals are defined on the server side.

Service discovery (kubernetes_sd, consul_sd, ec2_sd, file_sd) automatically tracks dynamic targets, and relabeling filters or rewrites labels.

Debugging is simple: curl http://target:port/metrics shows the raw metrics.

Pull excels in dynamic, large‑scale container environments, but it has hard limits:

Short‑lived jobs (e.g., a 5‑second CronJob) may finish before the next scrape, causing metric loss.

Serverless functions (Lambda, FaaS) disappear before a scrape can happen.

Targets behind NAT, firewalls, or at the edge may be unreachable from the central scraper.

At massive scale (hundreds of thousands of targets) the scraper’s fan‑out becomes a network bottleneck, requiring jitter and sharding.

To address short‑lived jobs, Prometheus offers Pushgateway, a “push‑to‑pull” bridge. Short tasks push metrics to Pushgateway; Prometheus then scrapes the gateway.

Pushgateway pitfalls:

Metrics never expire automatically; stale data must be deleted manually.

The up metric reflects Pushgateway health, not the health of the original short tasks.

It introduces a single point of failure.

Label conflicts (e.g., honor_labels misconfiguration) can cause metric overwrites.

Thus Pushgateway is suitable only for narrow use‑cases like batch result aggregation, not as a universal metric gateway.

Hybrid edge‑pull + central‑push architecture

In production at million‑QPS scale, most teams combine both paradigms: edge agents pull local targets, then push aggregated data to a central backend via remote_write. This retains pull’s flow‑control, health checks, and service discovery at the edge, while leveraging push’s network penetration and horizontal scalability for long‑term storage (Thanos, Mimir, VictoriaMetrics).

Key components:

Collection side (pull) : an agent runs on each node, scrapes local services, generates up health metrics, and performs relabeling.

Transport side (push) : the agent aggregates and pushes metrics using remote_write, which includes queueing, WAL for crash safety, and back‑pressure handling.

OpenTelemetry Collector embodies this dual mode: it can receive push from SDKs and pull from Prometheus receivers, then export uniformly.

Agent/DaemonSet patterns further distribute scraping, avoiding a central scraper’s fan‑out bottleneck.

Four dimensions for selecting a model

The decision can be broken into:

Target lifecycle (long‑running service vs short task)

Network topology (reachable endpoint vs NAT/edge)

Scale (number of targets, required fan‑out)

Security (who holds credentials, attack surface)

Guidelines:

Long‑running services → Pull (benefits from up health checks).

Short tasks / Serverless / edge → Push (no stable endpoint to scrape).

Cross‑cluster or long‑term storage → Push with remote_write.

Consider security: Pull requires only a read‑only endpoint, Push requires clients to hold write credentials.

Final takeaways

The earlier StatsD‑centric push model broke under load because the collector could not control traffic. Prometheus’s pull model solved many of those problems but cannot handle short‑lived or unreachable targets. The pragmatic solution for million‑QPS, ten‑thousand‑instance environments is a hybrid: edge agents pull locally, then push aggregated data centrally. This combines pull’s flow‑control and health detection with push’s network penetration and scalable storage.