Scaling Prometheus to Thousands of Nodes with Thanos: Architecture, Storage, and HA Practices

Background and challenges

In 2026, Kubernetes clusters of 3,000–5,000 nodes generate tens of thousands of Pods. A single Prometheus instance stores data locally in TSM files, which creates three primary bottlenecks at thousand‑node scale:

Storage capacity : 5,000 metrics scraped every 15 seconds produce ~5.76 GB per day; real workloads exceed 100 GB per day, exhausting local disks in weeks.

Query performance : When stored data exceeds 500 GB, the 90th‑percentile query latency for functions such as rate() and sum() degrades from milliseconds to seconds.

High‑availability & durability : A single Prometheus pod is a point of failure; replication doubles storage cost.

Architecture evolution

Federation

Multiple Prometheus instances scrape separate domains and a central Prometheus federates them. The central instance still bears all query load, stores data independently, and cannot perform cross‑instance historical queries.

Thanos (CNCF graduated)

Thanos adds a distributed layer on top of Prometheus without modifying Prometheus itself. Core components are:

Thanos Sidecar : runs as a Pod alongside Prometheus, reads the WAL and uploads TSM blocks to object storage.

Thanos Query : stateless gateway that aggregates results from Sidecars and Store Gateways.

Thanos Store Gateway : GRPC proxy that reads blocks from object storage.

Thanos Compactor : compresses blocks and creates down‑sampled data.

Thanos Ruler : evaluates recording and alerting rules, writing results back to object storage.

Thanos is preferred for clusters >1,000 nodes because its query scalability is linear and it provides long‑term storage.

Core design details

Data write path

Prometheus writes to local TSM files as usual. The Sidecar continuously tails the WAL, packages data into TSM blocks, and uploads them to the /store/pending path in object storage. After the Compactor finishes, blocks are moved to /store/compact for Store Gateway consumption.

Query path parallelisation

A PromQL query (e.g., sum(rate(http_requests_total{job="api"}[5m])) by (service)) is received by Thanos Query, which sends Info requests to all registered Sidecars and Store Gateways to discover relevant blocks. The query is then executed in parallel across those back‑ends, de‑duplicated using the replica label, and finally aggregated. Deploying multiple Query instances behind a Service load balancer provides near‑linear throughput increase.

Storage format & down‑sampling

The Compactor merges 2‑hour raw blocks and creates 5‑minute and 1‑hour down‑sampled series. Example: a 30‑day query on raw 15‑second data (≈170 M points) takes ~30 s, while the same query on 1‑hour down‑sampled data (≈720 points) returns in ~300 ms.

raw: retain 0‑30 days, 15 s resolution
5m: retain 30‑90 days
1h: retain >90 days

Block file structure

Each 2‑hour block contains chunk, index, meta.json, and tombstones. meta.json records ULID, time range, source instance, and down‑sampling level, which is essential for troubleshooting missing data.

Object store selection

Backend comparison

S3‑compatible (MinIO/AWS S3) – mature ecosystem, full toolchain; pay‑per‑request, cold‑vs‑hot cost variance; suitable for large‑scale, multi‑cloud deployments.

Google GCS – good GKE integration, read‑optimised; higher cross‑region latency; best for GCP native environments.

Azure Blob – very low cold‑storage cost; metadata operations slower; best for Azure environments.

Alibaba OSS – strong domestic compliance; requires VPC binding; best for Alibaba Cloud environments.

For thousand‑node clusters, S3‑compatible storage is recommended; in a pure Alibaba Cloud environment, OSS offers better cost‑performance despite higher request fees.

MinIO cluster deployment

# MinIO distributed cluster – 8 nodes, 4 disks each
# Erasure coding: data=6, parity=2
export MINIO_ROOT_USER=prometheus
export MINIO_ROOT_PASSWORD=ThanosSecure2026!
minio server http://minio{1...8}.cluster.local:9000/data{1...4} \
  --console-address ":9001" \
  --certs-dir /opt/minio/certs \
  --address ":9000"

The cluster provides ~5 GB/s aggregate write throughput, sufficient for the typical 100‑200 GB daily upload volume.

Thanos object‑store configuration

type: S3
config:
  bucket: thanos-data
  endpoint: minio.cluster.local:9000
  region: us-east-1
  access_key: thanos_uploader
  secret_key: Than0sSecret2026!
  s3_force_path_style: true
  http_config:
    idle_conn_timeout: 90s
    response_header_timeout: 120s
  part_size: 5242880   # 5 MB multipart upload
  signature_version2: false

Key tuning points: increase idle_conn_timeout to 120 s for long‑haul transfers, use a 5 MB part size for MinIO friendliness.

Query layer sharding – Receiver & Store Gateway

Thanos Receiver introduction

Standard Sidecar uploads only after a block is closed (≈2 h), causing a visibility gap. Receiver provides a remote_write endpoint that streams data directly to a local TSM store, which the Sidecar then uploads. This eliminates the “data visibility delay”.

# Data flow A (Sidecar): Prometheus → Sidecar → Object Store → Store Gateway
# Data flow B (Receiver): Prometheus → remote_write → Receiver → Query (real‑time)
#                               ↓
#                         Object Store (async)

Receiver cluster deployment

type: receive
http:
  listen: 10902
grpc:
  listen: 10901
remote-write:
  timeout: 30s
  queue_config:
    capacity: 100000
    max_shards: 50
    min_shards: 10
    max_samples_per_send: 10000
replication:
  factor: 2
labels:
  - name: cluster
    value: prod-cluster

Three Receiver instances with replication.factor=2 give double‑write redundancy. Assuming 1 M series per second, each point 200 bytes, the cluster can ingest ~13 MB/s; three nodes provide ~50 MB/s headroom.

Store Gateway caching

Two caches mitigate object‑store latency:

Metadata cache : stores block indexes; a 4 GB cache yields >90 % hit rate.

Chunk cache : stores actual block data; 16‑64 GB recommended on SSD.

thanos store \
  --data-dir=/var/lib/thanos/store \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --objstore.config-file=s3.yaml \
  --index-cache.json-file=/var/lib/thanos/index-cache.json \
  --store.lazy-streaming=true \
  --experimental.enable-streamed-snaphot-chunking

High availability & failure recovery

Prometheus HA deployment

Deploy two Prometheus replicas per scrape target, both using the same Sidecar. Thanos Query deduplicates using the replica label:

thanos query \
  --query.replica-labels=prometheus \
  --query.replica-labels=prometheus_replica

Use a PodDisruptionBudget and preStop hooks for graceful termination to avoid split‑brain when network quality differs.

Store Gateway availability

Run at least two Store Gateways with PodAntiAffinity so they schedule on different nodes. If all Store Gateways fail, Query can still serve recent data via Sidecars, but historical queries become unavailable.

Compactor failure & data repair

Only one Compactor instance runs at a time (distributed lock). If it crashes, blocks may remain in pending. Manual repair steps:

# Inspect inconsistent blocks
thanos tools bucket inspect --objstore.config-file=s3.yaml \
  --min-time=2026-01-01T00:00:00Z --max-time=2026-03-30T00:00:00Z
# Verify block integrity
thanos tools bucket verify --objstore.config-file=s3.yaml --id=01H2B5GXYZ123D2ZZZZZZZZZZZZZZZZ
# Optionally trigger compaction
thanos compact --objstore.config-file=s3.yaml \
  --data-dir=/var/lib/thanos/compact --consistency-delay=5m \
  --acceptMalformedIndex

Compactor should run hourly and finish within 45 minutes; longer runtimes indicate the need for bucket partitioning.

Capacity planning & resource allocation

Data volume estimation

Assuming 1,000 nodes, 10 Pods per node, 80 % sidecar coverage, and 500 metrics per sidecar, total series ≈ 5 M. At a 15‑second scrape interval and 200 bytes per sample, daily raw data ≈ 1.7 TB; after 30‑day retention and down‑sampling, total object‑store size ≈ 4 TB (≈12 TB physical with 3‑replica).

Resource sizing

# Thanos Query (2 replicas)
CPU: 16 cores each
Memory: 32 Gi each

# Thanos Store Gateway (4 replicas)
CPU: 8 cores each
Memory: 64 Gi each (incl. chunk cache)
Disk: 500 Gi SSD per instance

# Thanos Receiver (3 replicas)
CPU: 8 cores each
Memory: 16 Gi each
Disk: 200 Gi SSD

Key Query flags: --query.max-concurrent=20 (default) – increase to 50 under heavy load. --http-grace-period=5m – allows in‑flight queries to finish on SIGTERM.

Typical failure cases & troubleshooting

Store discovery failure

Symptom: UI shows “no stores”. Steps:

# Verify Store API
curl -s http://thanos-query.monitoring:10902/api/v1/stores | jq
# Check DNS resolution
kubectl exec -it thanos-query-0 -- nslookup stores.monitoring.svc.cluster.local
# Test GRPC port
kubectl exec -it thanos-query-0 -- nc -zv thanos-store-gateway.monitoring 10901
# Inspect Store logs for registration errors

Root cause is often a mismatched --grpc-address; use a headless Service with publishNotReadyAddresses=true.

Historical data missing

Symptom: queries older than 90 days return empty. Steps:

# List blocks in the time range
thanos tools bucket ls --objstore.config-file=s3.yaml \
  --min-time=2025-12-01T00:00:00Z --max-time=2025-12-31T23:59:59Z
# Inspect a specific block
thanos tools bucket inspect --objstore.config-file=s3.yaml --id=01H…
# Verify Compactor is running
kubectl logs -l app=thanos-compact | grep -i "compaction\|downsample"
# Manually verify Store access
thanos tools bucket verify --objstore.config-file=s3.yaml

Possible causes: Compactor never ran (blocks stay in pending), aggressive down‑sampling removed raw data, or object‑store lifecycle policies deleted blocks.

Prometheus OOM despite available memory

Symptom: OOM Killer terminates Prometheus while host memory is free. Diagnosis uses Prometheus metrics to locate high‑cardinality series and label explosion, especially from Istio sidecars. Example remediation steps:

# Identify high‑cardinality series
curl -s localhost:9090/metrics | grep prometheus_tsdb_head_series | awk '{if($2>1000000) print $0}'
# List series per job
for job in $(curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByMetricName[].name'); do
  count=$(curl -s "localhost:9090/api/v1/label/${job}/values" | jq '.data | length')
  echo "${job}: ${count} series"
done

Fix by reducing high‑cardinality labels via relabel_configs (e.g., drop trace_id, user_id), limiting remote_write queue capacity, and disabling unnecessary Istio metrics.

Best‑practice checklist

Architectural principles

Partition before scaling : split data by cluster, namespace, or job and keep each Thanos Query limited to a single bucket prefix.

Separate read/write paths : distinct Services for ingestion (Sidecar/Receiver) and query (Query/Store).

Remote‑write throttling : configure capacity, max_shards, and metadata_appended_timeout to apply back‑pressure.

Operational practices

Backup object‑store with cross‑region replication (CRR).

Deploy a minimal Prometheus instance to monitor Thanos components (GRPC latency, bucket operation duration, queue depth).

Retention policy: 0‑15 days raw (15 s), 15‑90 days 5 min down‑sample, >90 days 1 h down‑sample.

Performance optimisations

Use recording rules in Thanos Ruler for heavy aggregations.

Limit external labels to ≤5 and cardinality ≤100 each.

Adjust --query.max-concurrent based on observed query load.

Evidence chain & conclusion

Empirical data shows a single Prometheus instance with >500 GB data suffers >1 s 90th‑percentile query latency and frequent OOM due to high‑cardinality labels. Thanos’ Sidecar‑Query‑Store separation decouples ingestion from storage, enabling linear query scaling (≈40 % throughput increase per additional Query replica). Down‑sampling reduces a 30‑day query latency from 30 s to 300 ms while shrinking storage to ~1/1000 of raw size. Dual‑replica Prometheus plus object‑store CRR guarantees data durability across data‑center failures with RTO < 1 h.

A properly partitioned, tuned, and monitored Thanos‑based architecture provides a production‑grade solution for monitoring thousand‑node Kubernetes clusters in 2026.