Why Switch from Prometheus? Deploy a High‑Performance vmagent Cluster with VictoriaMetrics

Why Replace Prometheus?

In the cloud‑native era, metric collection is crucial for observability. Prometheus, while the de‑facto standard, suffers from a single‑node architecture and local storage that hinder scaling in large‑scale scenarios.

Limitations of Prometheus

Single‑node architecture: No native clustering, making horizontal scaling difficult.

Memory explosion: Memory usage spikes and frequent GC when targets exceed 100,000.

Unstable remote_write: Data loss under high load.

Lack of built‑in deduplication/buffering: Network jitter or backend failures cause metric loss.

To address these issues, we evaluated several solutions and chose vmagent from the VictoriaMetrics ecosystem as the next‑generation metric collector for production.

Deploying a High‑Availability vmagent Cluster

1. Create the vmagent root directory:

sudo mkdir -p /app/vmagent/{bin,tmp}
sudo chown -R ops. /app/vmagent

2. Download the vmagent binary:

$ wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.128.0/vmutils-linux-amd64-v1.128.0.tar.gz
$ tar xf vmutils-linux-amd64-v1.128.0.tar.gz -C /tmp/
$ cp /tmp/vmagent-prod /app/vmagent/bin/

3. Create the metrics scrape configuration:

cat <<'EOF' | tee /app/vmagent/scrape.yml > /dev/null
global:
  scrape_interval: 15s

scrape_configs:
- job_name: vmagent
  metrics_path: /metrics
  static_configs:
  - targets:
    - 172.139.20.181:8429
EOF

4. Define the systemd service:

cat <<'EOF' | sudo tee /lib/systemd/system/vmagent.service > /dev/null
[Unit]
Description=VictoriaMetrics vmagent service
After=network.target

[Service]
Type=simple
User=ops
Restart=always
ExecStart=/app/vmagent/bin/vmagent-prod -httpListenAddr=:8429 -promscrape.config=/app/vmagent/scrape.yml -remoteWrite.tmpDataPath=/app/vmagent/tmp -remoteWrite.url=http://172.139.20.200:18480/insert/0/prometheus
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target
EOF

5. Start and enable the service:

$ sudo systemctl daemon-reload
$ sudo systemctl enable vmagent --now

Verification

Check that data is not duplicated. With a 15‑second scrape interval, a 30‑second query should return two data points, confirming proper collection without duplication.

Conclusion

We do not dismiss Prometheus—it remains an excellent tool for learning and small‑scale scenarios. However, for enterprise‑grade, large‑scale, high‑reliability production environments, the vmagent + VictoriaMetrics combination offers superior performance, lower cost, and greater resilience. If Prometheus scaling is a pain point for you, trying vmagent could be the key upgrade step.