How to Build a Complete Longhorn Monitoring System with Prometheus & Grafana

In the cloud‑native era, Kubernetes is the standard for container orchestration and Longhorn is a lightweight, reliable distributed block storage solution. As workloads grow, effective monitoring and performance tuning of Longhorn become essential for operators.

This article walks through the core components of Longhorn and shows how to build a complete monitoring system using Prometheus and Grafana, from collecting metrics to setting up alerts.

1. Retrieve metrics data

The Longhorn metrics endpoint is exposed by the manager pod. Example commands:

<code>$ kubectl -n storage-system get pod -l app=longhorn-manager -o wide
NAME               READY   STATUS    RESTARTS   AGE   IP            NODE
longhorn-manager-94f2h   2/2   Running   11 (7h23m ago)   47h   10.244.135.133   k8s-node03
longhorn-manager-hf95p   2/2   Running   11 (7h22m ago)   47h   10.244.217.183   k8s-node04
longhorn-manager-m849h   2/2   Running   11 (7h23m ago)   47h   10.244.85.199    k8s-node01
longhorn-manager-x6jgl   2/2   Running   10 (7h23m ago)   47h   10.244.58.209    k8s-node02

$ curl -s 10.244.135.133:9500/metrics | head
# HELP longhorn_disk_capacity_bytes The storage capacity of this disk
# TYPE longhorn_disk_capacity_bytes gauge
longhorn_disk_capacity_bytes{disk="default-disk-fd1000000000",node="k8s-node03"} 1.073217536e+11
# HELP longhorn_disk_reservation_bytes The reserved storage for other applications and system on this disk
# TYPE longhorn_disk_reservation_bytes gauge
longhorn_disk_reservation_bytes{disk="default-disk-fd1000000000",node="k8s-node03"} 3.219652608e+10
# HELP longhorn_disk_status The status of this disk
# TYPE longhorn_disk_status gauge
longhorn_disk_status{condition="ready",condition_reason="",disk="default-disk-fd1000000000",node="k8s-node03"} 1
longhorn_disk_status{condition="schedulable",condition_reason="",disk="default-disk-fd1000000000",node="k8s-node03"} 1</code>

2. Configure Prometheus to scrape Longhorn

<code>job_name: 'longhorn'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
  source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_app,__meta_kubernetes_pod_container_port_name]
  regex: storage-system;longhorn-manager;manager</code>

3. Verify collection

<code>$ curl -s -u admin $(kubectl -n kube-system get svc prometheus -ojsonpath='{.spec.clusterIP}:{.spec.ports[0].port}')/prometheus/api/v1/query --data-urlencode 'query=up{job=~"longhorn"}' | jq '.data.result[] | {job: .metric.job, instance: .metric.instance ,status: .value[1]}'
{"job":"longhorn","instance":"10.244.85.199:9500","status":"1"}
{"job":"longhorn","instance":"10.244.135.133:9500","status":"1"}
{"job":"longhorn","instance":"10.244.58.209:9500","status":"1"}
{"job":"longhorn","instance":"10.244.217.183:9500","status":"1"}</code>

The query returns status “1” for each Longhorn manager instance, confirming successful scraping.

4. Add Grafana dashboard

Import dashboard ID 16888 (Longhorn) into Grafana to visualize the collected metrics.

Conclusion

Longhorn is increasingly adopted as a distributed storage solution on Kubernetes. Building a robust monitoring system is the “saddle” that lets its high availability and maintainability shine.