How to Build a High‑Availability Prometheus Monitoring System: Pitfalls & Performance Tuning

Building a High‑Availability Prometheus Monitoring System

Core value : This article shares complete production experience building a Prometheus monitoring system, including pitfalls, performance tuning, and best practices to help you avoid common mistakes and quickly set up an enterprise‑grade monitoring solution.

Why Choose Prometheus?

In the cloud‑native era, traditional monitoring tools cannot meet the complex needs of micro‑service architectures. Prometheus offers a pull model, multi‑dimensional data model, and a powerful query language (PromQL), making it the monitoring benchmark of CNCF projects.

Architecture Design: The Foundation of High Availability

Core Architecture Principle

Federated cluster mode is strongly recommended for production:

# Federation configuration example
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~"kubernetes-.*"}'
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets:
        - 'prometheus-shard1:9090'
        - 'prometheus-shard2:9090'

Sharding Strategy

Infrastructure sharding : monitor physical machines, network devices

Application sharding : divide by business line

Middleware sharding : databases, caches, message queues

Production Pitfall Guide

Pitfall 1: Uncontrolled Memory Usage

Symptom : Prometheus memory continuously grows and eventually OOM.

Root cause : High‑cardinality labels cause time‑series explosion.

# Check high‑cardinality labels
curl 'http://localhost:9090/api/v1/label/__name__/values' | jq '.data[]' | wc -l

# View number of series in memory
curl 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_symbol_table_size_bytes'

Solution :

# Limit label cardinality
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'high_cardinality_metric.*'
    action: drop
  - source_labels: [user_id]
    regex: '.*'
    target_label: user_id
    replacement: 'masked'

Pitfall 2: Query Performance Issues

Symptom : Complex queries time out, Grafana panels load slowly.

Root cause : Query time range too large, inefficient aggregation.

# ❌ Bad: large time‑range aggregation
rate(http_requests_total[1d])

# ✅ Good: use recording rules
job:http_requests:rate5m

Pitfall 3: Storage Space Problems

In production, storage growth often exceeds expectations.

# Storage optimization configuration
storage:
  tsdb:
    retention.time: 30d
    retention.size: 100GB
    min-block-duration: 2h
    max-block-duration: 36h

Performance Tuning Practice

Memory Tuning

Adjust JVM (if using Java) and system parameters according to monitoring scale:

# System‑level tuning
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf
echo 'fs.file-max=65536' >> /etc/sysctl.conf

# Prometheus startup parameters
./prometheus \
  --storage.tsdb.path=/data/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=100GB \
  --query.max-concurrency=20 \
  --query.max-samples=50000000

Recording Rules Optimization

Pre‑compute complex queries to improve performance:

groups:
  - name: http_requests
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)
      - record: job:http_requests_errors:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
      - record: job:http_requests_error_rate
        expr: job:http_requests_errors:rate5m / job:http_requests:rate5m

Storage Layer Optimization

Use remote storage for long‑term retention:

# Remote storage configuration
remote_write:
  - url: "http://thanos-receive:19291/api/v1/receive"
    queue_config:
      max_samples_per_send: 10000
      batch_send_deadline: 5s
      max_shards: 200

High‑Availability Deployment Practice

Multi‑Replica Deployment

# Kubernetes StatefulSet example
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
spec:
  replicas: 2
  selector:
    matchLabels:
      app: prometheus
  template:
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.45.0
          args:
            - '--storage.tsdb.path=/prometheus'
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--web.console.libraries=/etc/prometheus/console_libraries'
            - '--web.console.templates=/etc/prometheus/consoles'
            - '--web.enable-lifecycle'
            - '--web.enable-admin-api'
          resources:
            requests:
              memory: "4Gi"
              cpu: "1000m"
            limits:
              memory: "8Gi"
              cpu: "2000m"

Data Consistency Guarantee

Use Thanos for long‑term storage and global queries:

# Thanos Sidecar configuration
- name: thanos-sidecar
  image: thanosio/thanos:v0.31.0
  args:
    - sidecar
    - --tsdb.path=/prometheus
    - --prometheus.url=http://localhost:9090
    - --objstore.config-file=/etc/thanos/objstore.yml

Key Metrics Monitoring

Prometheus Self‑Monitoring

# TSDB metrics
prometheus_tsdb_head_series
prometheus_tsdb_head_samples_appended_total
prometheus_config_last_reload_successful

# Query performance metrics
prometheus_engine_query_duration_seconds
prometheus_engine_queries_concurrent_max

Alert Rule Design

groups:
  - name: prometheus.rules
    rules:
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus configuration reload failed"
      - alert: PrometheusQueryHigh
        expr: rate(prometheus_engine_query_duration_seconds_sum[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus query latency too high"

Fault Diagnosis Techniques

Common Troubleshooting Commands

# Check configuration syntax
./promtool check config prometheus.yml

# Check rule syntax
./promtool check rules /etc/prometheus/rules/*.yml

# View TSDB status
curl localhost:9090/api/v1/status/tsdb

# Analyze query performance
curl 'localhost:9090/api/v1/query?query=up&stats=all'

Performance Analysis Tools

Use Go's pprof to analyze Prometheus performance:

# Get CPU profile
go tool pprof http://localhost:9090/debug/pprof/profile

# Get memory profile
go tool pprof http://localhost:9090/debug/pprof/heap

Advanced Optimization Directions

1. Automatic Scaling

Implement auto‑scaling of Prometheus clusters based on query load and storage usage.

2. Intelligent Routing

Route queries to the most suitable Prometheus instance according to query patterns.

3. Machine‑Learning Optimization

Apply ML algorithms to predict resource demand and perform proactive capacity planning.

Conclusion

Building a highly available Prometheus monitoring system is a systematic engineering effort that requires careful architecture design, performance tuning, and fault handling across multiple dimensions. The practical experience and pitfall guide shared in this article aim to help you quickly set up a stable and reliable monitoring solution.