How to Backup and Restore VictoriaMetrics with vmbackup & vmrestore

Background

In a previous article we introduced vmstorage, vminsert, vmselect, vmui, and vmagent, which handle data storage, ingestion, querying, visualization, and collection. The missing piece is data backup and restore, which we cover here.

vmbackup

vmbackup is a backup tool that creates snapshots from vmstorage and can store backup data in various storage systems such as:

GCS (e.g., gs://<path/to/backup>)

S3 (e.g., s3://<path/to/backup>)

Any S3‑compatible storage like MinIO, Ceph, or Swift

Local filesystem (e.g., fs://</absolute/path/to/backup>)

vmbackup supports incremental and full backups. When a backup path already contains previous data, incremental backups run automatically; a full backup can use the -origin flag to accelerate copying of shared data between the new and existing backups. The backup process can be interrupted at any time and will resume from the breakpoint when the same parameters are used again. Data backed up with vmbackup can be restored using vmrestore.

vmrestore

vmrestore restores data from backups created by vmbackup. Like the backup process, restoration can be interrupted and resumed with the same command and parameters, replacing the original vmstorage data with the restored data.

Practical Steps

Data Backup

1. Create a snapshot

vmbackup requires an instant snapshot, which can be created via the vmstorage API:

# request
curl http://<victoriametrics-addr>:8428/snapshot/create # adjust address and port as needed

# response
{"status":"ok","snapshot":"<snapshot-name>"}

Example:

[root@test ~]# curl http://172.16.47.122:8482/snapshot/create
{"status":"ok","snapshot":"20220119051931-16CA0D126A870F8B"}

Check vmstorage logs to confirm snapshot creation:

2022-01-19T05:19:31.767Z info VictoriaMetrics/lib/storage/storage.go:304 creating Storage snapshot for "/vm-data"...
... (additional log lines) ...
2022-01-19T05:19:31.971Z info VictoriaMetrics/lib/storage/storage.go:361 created Storage snapshot for "/vm-data" at "/vm-data/snapshots/20220119051931-16CA0D126A870F8B" in 0.204 seconds

List snapshots via API:

# curl http://172.16.47.122:8482/snapshot/list
{"status":"ok","snapshots":["20220119051931-16CA0D126A870F8B"]}

2. Perform backup

Using the created snapshot, back up to a local filesystem (fs:///opt) as an example:

# ./vmbackup-prod -storageDataPath=/vm-data -snapshotName=20220119051931-16CA0D126A870F8B -dst=fs:///opt
... (log output) ...
2022-01-19T05:45:30.374Z info VictoriaMetrics/lib/backup/actions/backup.go:171 backup from src fslocal "/vm-data/snapshots/20220119051931-16CA0D126A870F8B" to dst fsremote "/opt" with origin fsnil is complete; backed up 125373885 bytes in 0.664 seconds; uploaded 125373885 bytes

Verify backup files in /opt.

3. Create a second snapshot

# curl http://172.16.47.122:8482/snapshot/create
{"status":"ok","snapshot":"20220119055254-16CA0D126A870F8C"}

4. Incremental backup

# ./vmbackup-prod -storageDataPath=/vm-data -snapshotName=20220119055254-16CA0D126A870F8C -dst=fs:///opt
... (log) ...
2022-01-19T05:55:18.548Z info VictoriaMetrics/lib/backup/actions/backup.go:171 backup from src fslocal "/vm-data/snapshots/20220119055254-16CA0D126A870F8C" to dst fsremote "/opt" with origin fsnil is complete; backed up 127597222 bytes in 0.214 seconds; deleted 2714902 bytes; uploaded 4938239 bytes

The non‑zero delete value shows files present in the first snapshot but absent in the second, and the operation is faster than the full backup.

5. Create another snapshot to test server‑side copy

# curl http://172.16.47.122:8482/snapshot/create
{"status":"ok","snapshot":"20220119073821-16CA0D126A870F8D"}

6. Full backup with -origin to enable server‑side copy

# ./vmbackup-prod -storageDataPath=/vm-data -snapshotName=20220119073821-16CA0D126A870F8D -dst=fs:///ss -origin=fs:///opt
... (log) ...
2022-01-19T07:44:56.657Z info VictoriaMetrics/lib/backup/actions/backup.go:171 backup from src fslocal "/vm-data/snapshots/20220119073821-16CA0D126A870F8D" to dst fsremote "/ss" with origin fsremote "/opt" is complete; backed up 131747249 bytes in 0.190 seconds; server-side copied 121316947 bytes; uploaded 10430302 bytes

Data Restore

Restore the backup from fs:///ss to a new storage path /tt:

# ./vmrestore-prod -src=fs:///ss -storageDataPath=/tt
... (log) ...
2022-01-19T07:58:32.751Z info VictoriaMetrics/lib/backup/actions/restore.go:185 downloaded 131747249 out of 131747249 bytes from fsremote "/ss" to fslocal "/tt" in 624.892637ms
2022-01-19T07:58:32.751Z info VictoriaMetrics/lib/backup/actions/restore.go:192 restored 131747249 bytes from backup in 0.635 seconds

After restoration, stop the VM, then restart it with -storageDataPath=/tt to use the restored data.

Troubleshooting

If backup is slow, increase concurrency with the -concurrency flag.

Limit bandwidth usage with -maxBytesPerSecond if vmbackup/vmrestore consumes too much network.

If a backup or restore is interrupted, rerun the same command; it will resume from the last checkpoint.

Backups from a single‑node VM cannot be used to restore data to a clustered VM.

Summary

This article introduced vmbackup and vmrestore, demonstrating how to create snapshots, perform incremental and full backups to various storage backends, and restore data, with practical command examples and troubleshooting advice.

Reference

https://docs.victoriametrics.com