Prometheus High Availability: Thanos vs VictoriaMetrics Deep Dive & Selection Guide

Problem Background

Single-node Prometheus fails under high cardinality: a cluster with 50 nodes and 500 pods saw series grow from 800k to 9M after adding custom metrics, causing OOM kills and 25-minute WAL replay downtime. Running two Prometheus instances does not provide true HA because data diverges and alerts duplicate.

True HA requires solving three layers: query HA (dashboards/alerts survive instance failure), storage extension (long-term retention beyond local disk), and scrape HA (collection continues despite target failures, no duplicate alerts).

Architecture Comparison

Thanos

Thanos wraps Prometheus without modifying it. Components: Sidecar (uploads blocks to object storage, serves local data), Store Gateway (reads blocks from object storage), Querier (aggregates StoreAPI endpoints), Compactor (downsampling, retention, single instance), Ruler (alerting/recording rules), Receive (accepts remote write with replication), Query Frontend (query splitting, caching). Two modes: Sidecar (each Prometheus keeps local storage, Sidecar uploads) and Receive (Prometheus remote-writes to Receive, which replicates). Sidecar suits sharded scraping; Receive suits redundant scraping of same targets.

VictoriaMetrics

VM uses its own storage engine (mergeset), not Prometheus TSDB. Single-node binary ( victoria-metrics) handles ingestion, storage, querying. Cluster version splits into vminsert (write path, stateless), vmstorage (data storage, stateful, no auto-rebalance), vmselect (query path, stateless), vmagent (scraping + remote write), vmalert (rules). Data organized in time-partitioned directories (daily/monthly), enabling fast deletion but making retention reduction irreversible.

Key Technical Differences

Storage format: Thanos reuses Prometheus TSDB blocks (2h immutable blocks uploaded to object storage). VM uses mergeset with columnar compression.

Long-term storage: Thanos requires object storage (S3/OSS/Ceph). VM stores on local disks (NVMe) with optional object storage backup.

Downsampling: Thanos Compactor generates 5m/1h downsampled blocks. VM queries raw data directly, no downsampling needed.

Replication: Thanos Sidecar mode has no replication; Receive mode replicates via hashring. VM replicates at vminsert via -replicationFactor.

Query language: Thanos uses native PromQL. VM uses MetricsQL (superset with WITH, limit, if, enhanced functions).

Compression: VM achieves ~0.48 bytes/sample vs Thanos ~0.95 bytes/sample (2x better). Advantage shrinks with high label cardinality (e.g., request_id).

Restart recovery: Prometheus+Thanos needs WAL replay (minutes). VM restarts in seconds (no WAL).

Version Matrix (as of Nov 2026)

Prometheus 3.16.0 (latest), 3.13.5 LTS (recommended for production).

Thanos v0.42.5 (note: removed --debug.advertise-compatibility-label and --shipper.ignore-unequal-block-size).

VictoriaMetrics v1.156.0 (monthly), v1.148.9 LTS (12-month support). Skip v1.154.0 (known issues).

Kubernetes 1.31–1.34 (tested on 1.33).

Decision Tree

Daily samples < 200M → single-node VM or Prometheus + large disk.

Need raw retention > 90 days? If no → single-node VM.

Have object storage and ops capability? Yes → Thanos (storage cost 5-10x cheaper). No → VM.

Multi-tenancy > 50 tenants → VM cluster (native accountID).

Require 100% PromQL compatibility → Thanos.

Team size 1-2 → VM (fewer components). 3+ with object storage skills → Thanos.

Large existing Prometheus rules/dashboards → both need validation; Thanos migration cheaper.

Cost Model (8M series, 15s scrape, 4.6B samples/day)

Thanos: 30d raw ~1.31TB (object storage), 5m downsample 180d ~0.13TB, 1h downsample 3yr ~0.40TB, local SSD 3TB, total object storage ~1.84TB.

VM cluster: 90d raw ~0.66TB (local SSD ×2 replication = 1.32TB), 3yr raw ~7.9TB (local SSD).

Object storage (~¥0.12/GB/mo) vs SSD (~¥1.5/GB/mo) makes Thanos cheaper for >1yr retention despite 2x larger raw size.

Operational Burden

Thanos

7 component types, each with config/metrics.

Compactor single point: halts on downsampling failures (e.g., S3 timeout), requires manual restart.

Store Gateway cache/index tuning ( --index-cache-size, --chunk-pool-size).

Object storage permissions, lifecycle, versioning.

Receive hashring changes cause brief write disruption.

VictoriaMetrics

3 component types, centralized config.

vmstorage cannot rebalance; new nodes only receive new data.

vmalert -replicaCount misconfiguration causes silent duplicate/missing alerts. -dedup.minScrapeInterval misconfiguration silently drops data.

Team of 1-2 → VM easier to run quietly. Dedicated SRE + object storage experience → Thanos offers richer ecosystem and zero-rule-migration.

Hybrid Option

Prometheus (15d retention, runs rules) → remote write → VM cluster (1yr retention). Pros: zero rule changes, cheap long-term storage, fast short-term queries. Cons: two systems to monitor; Prometheus remote write lacks persistent disk queue (data loss if VM down >2h). Suitable for <5M series, <1yr retention, no object storage desire.

Deployment Guides

Thanos Sidecar Mode

Prepare S3-compatible bucket, credentials via Secret.

Define stable external labels ( cluster, prometheus_replica) — immutable once written to block meta.json.

Inject Sidecar container into Prometheus StatefulSet with --shipper.upload-compacted=false, --prometheus.url=http://127.0.0.1:9090.

Set Prometheus --storage.tsdb.max-block-duration=2h to prevent local compaction delaying uploads.

Deploy Querier (2+ replicas) with --query.replica-label=prometheus_replica, --query.auto-downsampling.

Deploy Compactor (single replica) with --consistency-delay=30m (adjust based on upload P99), retention policies, --delete-delay=48h.

Verify: Store health, 7d query, downsample block existence, deduplication.

VictoriaMetrics Cluster

Validate disk IOPS (>5k, p99<5ms), file descriptors (>65k), clock sync (<100ms).

Deploy via Helm chart victoria-metrics-cluster.

Configure vmstorage: retentionPeriod=90d, memory.allowedPercent=60 with memory limit, storage.minFreeDiskSpaceBytes=100GiB.

vminsert: replicationFactor=2, dedup.minScrapeInterval=15s (match scrape interval).

vmagent: reuse Prometheus scrape_config, add -remoteWrite.tmpDataPath and -remoteWrite.maxDiskUsagePerURL=100GiB for disk queue.

vmalert: enable -clusterModeEnabled with -replicaCount=2 for sharded rule evaluation.

Verify: all vmstorage reachable, zero ignored rows, ingestion/query parity, merge health, snapshot creation.

Troubleshooting Patterns

Thanos missing history: Check Store Gateway sync, downsample block existence, --query.auto-downsampling.

VM slow queries: Distinguish systemic (merge contention, cache miss) vs query-specific (wide label matcher).

Data doubling: Thanos missing --query.replica-label matching external_labels; VM dedup.minScrapeInterval not set to scrape interval.

Write backlog: VM: check vmstorage reachable, slow inserts, disk IO. Thanos Receive: check forward errors, hashring consistency.

Capacity Planning Formulas

Thanos: daily_raw_bytes × raw_days × replicas + daily_raw/3 × 5m_days + daily_raw/12 × 1h_days + 10% compactor workspace.

VM: daily_bytes × retention_days × replication_factor × 1.5 (merge overhead).

Example: 180GB/day raw → Thanos ~22TB object storage (3yr), VM ~13.5TB local (90d) or ~164TB local (3yr).

Backup & Recovery

Thanos: enable object storage versioning (30d non-current), lifecycle only transitions storage class (no expiration), quarterly cross-account read test.

VM: daily vmbackup snapshots to object storage (14d retention), quarterly vmrestore drill in isolated env.

Key Takeaways

Don't over-engineer HA for <5M series; single VM + daily backup + fast rebuild often more reliable.

Compactor halt and VM retention reduction are top data-loss risks; enforce change approval, backups, two-person review.

Scrape layer doesn't need HA; write layer needs replication with deduplication.

Monitor the monitoring: alert on thanos_compact_halted, vm_merge_need_free_disk_space — silent data gaps don't trigger business alerts.

Keep old system read-only for weeks after migration.

Archive benchmark data; future scaling decisions depend on your own measured P99, not community claims.