Deploy and Configure vmagent on Kubernetes for Efficient Metrics

Background

In a previous article we explained how to install VictoriaMetrics (vm) on Kubernetes, using OpenTelemetry for metric collection. vm also provides its own agent, vmagent, which can replace OpenTelemetry for gathering metrics and storing them in vmstorage.

What is vmagent

Official description:

<code>vmagent is a tiny but mighty agent which helps you collect metrics from various sources and store them
in VictoriaMetrics or any other Prometheus‑compatible storage systems that support the remote_write protocol</code>

vmagent is a lightweight yet powerful agent that collects metrics from multiple sources and writes them to VictoriaMetrics or any Prometheus‑compatible remote_write endpoint. The official architecture diagram is shown below.

Key Features of vmagent

Can act as a temporary Prometheus replacement, e.g., scrape data from node_exporter.

Can read from and write to Kafka.

Supports relabeling, adding, deleting, and modifying labels before sending data to remote storage.

Accepts data via the vm data injection protocol.

Can duplicate collected data to multiple remote storage systems.

Consumes less CPU, memory, disk I/O and network bandwidth than Prometheus.

…

How to Install vmagent

Prerequisites

Use the same environment as the previous article: keep vmcluster and node‑exporter, stop OpenTelemetry, and clean existing vm data so that the cluster starts without metrics. Deploy vmagent directly on Kubernetes.

Step 1 – Create Namespace and RBAC

<code># Create namespace
kubectl create ns monitoring-system

# Apply RBAC (yaml can be downloaded from the previous article)
kubectl apply -f vmagent_rbac.yaml

# vmagent_rbac.yaml (excerpt)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vmagent
  namespace: monitoring-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmagent
rules:
- apiGroups: ["","networking.k8s.io","extensions"]
  resources: ["nodes","nodes/metrics","services","endpoints","endpointslices","pods","app","ingresses"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["namespaces","configmaps"]
  verbs: ["get"]
- nonResourceURLs: ["/metrics","/metrics/resources"]
  verbs: ["get"]
- apiGroups: ["route.openshift.io","image.openshift.io"]
  resources: ["routers/metrics","registry/metrics"]
  verbs: ["get"]
...</code>

Step 2 – Deploy vmagent

<code># Install vmagent
kubectl apply -f vmagent.yaml

# vmagent.yaml (excerpt)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: vm-vmagent
  namespace: monitoring-system
spec:
  selectAllByDefault: true
  podMetadata:
    labels:
      victoriaMetrics: vmagent
  replicaCount: 1
  serviceAccountName: vmagent
  image:
    pullPolicy: IfNotPresent
    repository: images.huazai.com/release/vmagent
    tag: v1.63.0-cluster
  resources:
    requests:
      cpu: "250m"
      memory: "350Mi"
    limits:
      cpu: "500m"
      memory: "850Mi"
  extraArgs:
    memory.allowedPercent: "40"
  remoteWrite:
  - url: "http://lb_vip:8480/insert/0/prometheus/api/v1/write"
</code>

Step 3 – Verify Installation

<code># Check pod status
kubectl get po -n monitoring-system

# Example output
NAME                                 READY   STATUS    RESTARTS   AGE
vmagent-vm-vmagent-55bbbd9f6d-497m5   2/2     Running   4          2d17h
</code>

The pod contains two containers; the second one refreshes vm configuration.

Default Scrape Configuration

<code>global:
  scrape_interval: 30s
  external_labels:
    prometheus: monitoring-system/vm-vmagent
scrape_configs:
- job_name: monitoring-system/vmagent-vm-vmagent/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names: [monitoring-system]
  metrics_path: /metrics
  relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_component]
    regex: monitoring
  - action: keep
    source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance]
    regex: vm-vmagent
  - action: keep
    source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
    regex: vmagent
  - action: keep
    source_labels: [__meta_kubernetes_service_label_managed_by]
    regex: vm-operator
  - action: keep
    source_labels: [__meta_kubernetes_endpoint_port_name]
    regex: http
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind,__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    replacement: $1
    target_label: node
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind,__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    replacement: $1
    target_label: pod
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod
  - source_labels: [__meta_kubernetes_namespace]
    target_label: namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: service
  - source_labels: [__meta_kubernetes_service_name]
    target_label: job
    replacement: $1
  - target_label: endpoint
    replacement: http
</code>

Adding Custom Scrape Configs

To reuse the previous OpenTelemetry configuration, add it via

inlineScrapeConfig

:

<code># Re‑install vmagent with inlineScrapeConfig
kubectl apply -f vmagent.yaml

# vmagent.yaml (excerpt)
...
inlineScrapeConfig: |
  # Prometheus test job
  - job_name: "prometheus"
    static_configs:
    - targets: ["ip1:9100","ip2:9100"]
  - job_name: coredns
    kubernetes_sd_configs:
    - namespaces:
        names: [kube-system]
      role: endpoints
    relabel_configs:
    - action: keep
      regex: coredns;metrics
      source_labels: [__meta_kubernetes_service_name,__meta_kubernetes_endpoint_port_name]
    - source_labels: [__meta_kubernetes_pod_node_name]
      target_label: node
    - source_labels: [__meta_kubernetes_pod_host_ip]
      target_label: host_ip
  # (additional jobs for kube‑state‑metrics, node‑exporter, kube‑apiserver, etc.)
  ...
</code>

Configuration Refresh

Two ways to refresh vmagent configuration:

Send a SIGHUP signal to the vmagent process.

Send an HTTP request to

http://vmagent:8429/-/reload

.

Monitoring vmagent

vmagent exposes its own metrics at

http://vmagent-host:8429/metrics

, such as

vmagent_remotewrite_conns

(remote write connections) and

vm_allowed_memory_bytes

(available memory). These can be visualized in Grafana. Important API endpoints:

http://vmagent-host:8429/targets

– view scrape targets status.

http://vmagent-host:8429/ready

– check readiness.

Troubleshooting Tips

Monitor vmagent metrics to understand its health.

When scraping a large number of targets, increase the maximum number of open files on the host.

If the remote storage returns 400 or 409, vmagent discards the affected data blocks.

Refer to the official documentation for more details.

Conclusion

Installation is straightforward; just pay attention to the format of the

inlineScrapeConfig

field and the image used for the configuration‑refresh sidecar. This article provides a basic overview of vmagent; explore additional features on your own and give it a try.

References

https://docs.victoriametrics.com/vmagent.html

https://github.com/VictoriaMetrics/operator/blob/master/docs/quick-start.MD#VMAgent

https://github.com/VictoriaMetrics/operator/blob/master/docs/api.MD#vmagentspec

https://prometheus.io/