How to Build a Production‑Ready, High‑Availability VictoriaMetrics Cluster

In today’s cloud‑native and micro‑service era, observability is essential for system stability, and VictoriaMetrics provides a high‑performance, low‑resource, horizontally‑scalable alternative to Prometheus for metric monitoring.

The article walks through deploying a production‑grade, high‑availability VictoriaMetrics cluster, ensuring fault‑tolerance, scalability, and persistence.

VictoriaMetrics cluster architecture

The cluster consists of three core components:

vmstorage : stores time‑series data (TSDB) – stateful.

vminsert : receives Remote Write requests and forwards them to vmstorage – stateless.

vmselect : handles PromQL queries, aggregates data from vmstorage – stateless.

Step 0: Download and extract VictoriaMetrics

$ wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.128.0/victoria-metrics-linux-amd64-v1.128.0-cluster.tar.gz

$ tar xvf victoria-metrics-linux-amd64-v1.128.0-cluster.tar.gz -C /tmp/
vminsert-prod
vmselect-prod
vmstorage-prod

Step 1: Deploy vmstorage

# Create vmstorage directories
sudo mkdir -p /app/vmstorage/{data,bin}
sudo chown -R ops. /app/vmstorage

# vmstorage configuration (vmstorage.conf)
httpListenAddr=:8482
vminsertAddr=:8400
vmselectAddr=:8401
storageDataPath=/app/vmstorage/data
retentionPeriod=7d
dedup_minScrapeInterval=10s
loggerTimezone=Asia/Shanghai
loggerLevel=INFO
loggerFormat=default

# systemd service file (/usr/lib/systemd/system/vmstorage.service)
[Unit]
Description=VictoriaMetrics vmstorage service
After=network.target

[Service]
Type=simple
User=ops
Restart=always
EnvironmentFile=/app/vmstorage/vmstorage.conf
ExecStart=/app/vmstorage/bin/vmstorage-prod -envflag.enable
WorkingDirectory=/app/vmstorage
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target

# Start service
sudo systemctl daemon-reload
sudo systemctl start vmstorage --now

Tip: All vmstorage nodes should share the same configuration; at least three nodes are recommended.

Step 2: Deploy vminsert

# Create vminsert directory
sudo mkdir -p /app/vminsert/bin
sudo chown -R ops. /app/vminsert
cp /tmp/vminsert-prod /app/vminsert/bin/vminsert-prod

# vminsert configuration (vminsert.conf)
httpListenAddr=:8480
storageNode=172.139.20.17:8400,172.139.20.81:8400,172.139.20.177:8400
replicationFactor=2
loggerTimezone=Asia/Shanghai
loggerLevel=INFO
loggerFormat=default

# systemd service file (/usr/lib/systemd/system/vminsert.service)
[Unit]
Description=VictoriaMetrics vminsert service
After=network.target

[Service]
Type=simple
User=ops
Restart=always
EnvironmentFile=/app/vminsert/vminsert.conf
ExecStart=/app/vminsert/bin/vminsert-prod -envflag.enable
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target

# Start service
sudo systemctl daemon-reload
sudo systemctl start vminsert --now

Tip: All vminsert nodes share the same configuration; at least two nodes are recommended.

Step 3: Deploy vmselect

# Create vmselect directories
sudo mkdir -p /app/vmselect/{bin,cache}
sudo chown -R ops. /app/vmselect
cp /tmp/vmselect-prod /app/vmselect/bin/vmselect-prod

# vmselect configuration (vmselect.conf)
httpListenAddr=:8481
storageNode=172.139.20.17:8401,172.139.20.81:8401,172.139.20.177:8401
cacheDataPath=/app/vmselect/cache
vmui_defaultTimezone=Asia/Shanghai
loggerTimezone=Asia/Shanghai
loggerLevel=INFO
loggerFormat=default

# systemd service file (/usr/lib/systemd/system/vmselect.service)
[Unit]
Description=VictoriaMetrics vmselect service
After=network.target

[Service]
Type=simple
User=ops
Restart=always
EnvironmentFile=/app/vmselect/vmselect.conf
ExecStart=/app/vmselect/bin/vmselect-prod -envflag.enable
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target

# Start service
sudo systemctl daemon-reload
sudo systemctl start vmselect --now

Tip: All vmselect nodes share the same configuration; at least two nodes are recommended.

Step 4: Configure load balancing with HAProxy

listen vminsert
    bind *:18480
    mode tcp
    balance roundrobin
    server vminsert01 172.139.20.182:8480 check
    server vminsert02 172.139.20.183:8480 check

listen vmselect
    bind *:18481
    mode tcp
    balance roundrobin
    server vmselect01 172.139.20.182:8481 check
    server vmselect02 172.139.20.183:8481 check

Verification

Conclusion

Although binary deployment may lack the flashiness of containerization, it offers unmatched stability, low resource overhead, and efficient troubleshooting. By following these steps you can quickly build a production‑ready, highly available, and scalable VictoriaMetrics cluster on bare metal or virtual machines.