Node Exporter Metrics Explained: CPU, Memory, Disk & Network Monitoring

Verify the collection pipeline before tweaking thresholds

Node Exporter listens on TCP port 9100 and exposes metrics at /metrics. Any failure in the scrape chain (Prometheus target, service, network, or exporter) can cause a graph to show zero values, which does not mean the host resources are zero. First check the exporter version, startup parameters, and listening address using node_exporter --version, node_exporter --help, and

systemctl show node_exporter -p ExecStart -p FragmentPath -p User

. If managed by systemd, inspect its status and recent logs with systemctl status node_exporter --no-pager and

journalctl -u node_exporter --since '30 minutes ago' --no-pager

. Verify the HTTP endpoint locally with

curl --fail --silent --show-error --max-time 5 http://127.0.0.1:9100/metrics | sed -n '1,30p'

and ensure the output contains # HELP and # TYPE lines.

Metric type determines the query method

Node Exporter mainly provides counter and gauge metrics. Counters (e.g., node_cpu_seconds_total, node_disk_read_bytes_total, node_network_receive_bytes_total) are monotonic and should be queried with rate() or increase(). Gauges (e.g., node_memory_MemAvailable_bytes, node_load1) represent current values and can be compared directly or used in ratio calculations. Always confirm the metric type via its # TYPE metadata before writing queries.

CPU: split time then diagnose the cause of load

The metric node_cpu_seconds_total{cpu,mode} records cumulative seconds per CPU per mode (user, system, idle, iowait, irq, softirq, steal, nice). Overall CPU usage is derived from non‑idle time:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{job="node-exporter",mode="idle"}[5m])))

This answers "how busy the CPU is" but not "why". To pinpoint the cause, break down by mode:

100 * avg by (instance, mode) (rate(node_cpu_seconds_total{job="node-exporter",mode=~"user|system|iowait|steal|softirq"}[5m]))

user

high – application‑level CPU consumption. system high – kernel work, context switches, I/O stack. iowait high – CPU waiting for block device I/O (not disk utilization). steal high – vCPU stolen by the hypervisor. softirq high – network packet processing pressure.

Normalize by core count using

count by (instance) (node_cpu_seconds_total{job="node-exporter",mode="idle"})

. Compare load averages ( node_load1, node_load5, node_load15) against core count to detect I/O‑bound load (high load, low CPU usage). Use uptime, vmstat, and ps to correlate D‑state tasks and context‑switch rates ( rate(node_context_switches_total[5m]), rate(node_forks_total[5m])).

Memory: use MemAvailable instead of free

Linux repurposes idle memory for page cache, so node_memory_MemFree_bytes is often low. The more accurate indicator of usable memory is node_memory_MemAvailable_bytes, which estimates memory that can be allocated without swapping. Compute usage percentage with:

100 * (1 - node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"})

If the kernel lacks the MemAvailable metric, fall back to summing MemFree, Buffers, and Cached from /proc/meminfo. Monitor swap usage and swap activity:

100 * (1 - node_memory_SwapFree_bytes{job="node-exporter"} / node_memory_SwapTotal_bytes{job="node-exporter"})
rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m])
rate(node_vmstat_pgmajfault[5m])

Cross‑check with host tools ( free -h, cat /proc/meminfo, /proc/<PID>/smaps_rollup) and kernel logs for OOM events.

Disk: capacity, inode usage, and device performance

Filesystem capacity is exposed via node_filesystem_*, while device performance metrics come from node_disk_*. Calculate usable space excluding pseudo‑filesystems:

100 * (1 - node_filesystem_avail_bytes{job="node-exporter",fstype!~"tmpfs|overlay|squashfs|proc|sysfs"} / node_filesystem_size_bytes{job="node-exporter",fstype!~"tmpfs|overlay|squashfs|proc|sysfs"})

For user‑visible alerts use avail (excludes reserved blocks) rather than free. Inode usage is monitored separately:

100 * (1 - node_filesystem_files_free{job="node-exporter",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_files{job="node-exporter",fstype!~"tmpfs|overlay|squashfs"})

When space is sufficient but inodes are exhausted, file creation still fails. Verify mount points, filesystem type, and reserved blocks with findmnt, df -hT, and df -ih. To locate large directories without crossing mount boundaries, use a low‑priority du command:

ionice -c 3 nice -n 19 du -x -d 1 -h <mountpoint> 2>/dev/null | sort -h

Check for deleted but still‑open files with lsof +L1 <mountpoint>. Device throughput is derived from counters:

rate(node_disk_read_bytes_total{job="node-exporter",device!~"loop.*|ram.*"}[5m]) +
rate(node_disk_written_bytes_total{job="node-exporter",device!~"loop.*|ram.*"}[5m])

Average read latency (ms):

1000 * rate(node_disk_read_time_seconds_total[5m]) / clamp_min(rate(node_disk_reads_completed_total[5m]), 1)

Device busy percentage:

100 * rate(node_disk_io_time_seconds_total{job="node-exporter",device="<device>"}[5m])

Validate with iostat -xz 1 10 <device>, lsblk, and findmnt. For traditional single‑queue disks, near‑100% busy indicates saturation; for NVMe, RAID, or cloud disks, combine busy time with IOPS, latency, and queue depth.

Network: bandwidth is just the starting point

Network byte counters are counters; compute bits per second:

8 * sum by (instance, device) (rate(node_network_receive_bytes_total{job="node-exporter",device!="lo"}[5m]))
8 * sum by (instance, device) (rate(node_network_transmit_bytes_total{job="node-exporter",device!="lo"}[5m]))

To assess utilization, you need the interface speed, which may be missing for virtual NICs. Check errors and drops:

sum by (instance, device) (rate(node_network_receive_errs_total{job="node-exporter",device!="lo"}[5m]) + rate(node_network_transmit_errs_total{job="node-exporter",device!="lo"}[5m]))
sum by (instance, device) (rate(node_network_receive_drop_total{job="node-exporter",device!="lo"}[5m]) + rate(node_network_transmit_drop_total{job="node-exporter",device!="lo"}[5m]))

Zero counters do not guarantee loss‑free paths; packet loss can occur in switches, virtual bridges, or the TCP stack. Inspect the host stack with ip -s link show dev <iface>, ethtool -S <iface>, and nstat -az. TCP retransmission rate:

rate(node_netstat_Tcp_RetransSegs{job="node-exporter"}[5m]) / clamp_min(rate(node_netstat_Tcp_OutSegs{job="node-exporter"}[5m]), 1)

Socket pressure can be seen via ss -s, ss -lntp, and nstat output.

Collector management and secure deployment

Default collectors cover most basics. Before enabling extra collectors, verify version support, collection cost, and required permissions. The textfile collector is useful for custom host state; ensure writes are atomic to avoid half‑written files being scraped.

Example script writes a custom metric custom_root_readonly and a timestamp to a .prom file in the textfile directory. The script uses set -euo pipefail, creates a temporary file, writes HELP and TYPE lines, and atomically moves the file into place.

#!/usr/bin/env bash
set -euo pipefail

TEXTFILE_DIR="<textfile_dir>"
TARGET="${TEXTFILE_DIR}/host_checks.prom"
TMP_FILE="$(mktemp "${TEXTFILE_DIR}/.host_checks.XXXXXX")"
trap 'rm -f "${TMP_FILE}"' EXIT

readonly_root=0
findmnt -n -o OPTIONS / | tr ',' '
' | grep -qx ro && readonly_root=1
{
  echo '# HELP custom_root_readonly Whether root filesystem is read-only.'
  echo '# TYPE custom_root_readonly gauge'
  printf 'custom_root_readonly %s
' "${readonly_root}"
  printf 'custom_check_timestamp_seconds %s
' "$(date +%s)"
} > "${TMP_FILE}"
chmod 0644 "${TMP_FILE}"
mv -f "${TMP_FILE}" "${TARGET}"
trap - EXIT

Run the exporter as a low‑privilege systemd user, limit the listening address, and back up the unit file before changes. Sample unit file shows User=node-exporter, Group=node-exporter, and hardening options ( NoNewPrivileges=true, ProtectSystem=strict, ReadWritePaths=/var/lib/node_exporter/textfile_collector).

Deployment changes should be validated with systemd-analyze verify, a short‑lived test instance on an alternate port, and a curl check. After verification, reload systemd, restart the service, and confirm up{job="node-exporter"}=1 and scrape latency in Prometheus.

Alert rules: symptom, duration, and actionable context

Never use instantaneous thresholds alone. Example rule set groups alerts for exporter down, sustained CPU busy (>90% for 15 min), low available memory (<10% for 10 min), low filesystem space (<10% for 15 min), and low inode count (<10% for 15 min). Labels include severity; annotations provide a human‑readable summary.

groups:
- name: node-exporter-baseline
  rules:
  - alert: NodeExporterDown
    expr: up{job="node-exporter"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Node Exporter cannot scrape {{ $labels.instance }}"
  - alert: HostCpuBusy
    expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{job="node-exporter",mode="idle"}[5m]))) > 90
    for: 15m
    labels:
      severity: warning
  - alert: HostMemoryAvailableLow
    expr: node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} < 0.10
    for: 10m
    labels:
      severity: warning
  - alert: FilesystemSpaceLow
    expr: node_filesystem_avail_bytes{job="node-exporter",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="node-exporter",fstype!~"tmpfs|overlay|squashfs"} < 0.10
    for: 15m
    labels:
      severity: warning
  - alert: FilesystemInodesLow
    expr: node_filesystem_files_free{job="node-exporter",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_files{job="node-exporter",fstype!~"tmpfs|overlay|squashfs"} < 0.10
    for: 15m
    labels:
      severity: warning

Thresholds are just baselines; adjust per workload (databases, caches, VMs). Use predict_linear for capacity forecasting when trends are stable, and validate rule syntax with promtool check rules and promtool check config. After rule changes, reload Prometheus (HUP or /-/reload) and monitor prometheus_rule_evaluation_failures_total for errors.

Reusable investigation workflow

When a "host lag" alert fires, run a low‑risk diagnostic script that collects uname -a, uptime, free -h, vmstat, df -hT, ip -s link, ss -s, top processes, and recent kernel logs. The workflow then proceeds step‑by‑step: verify up and scrape freshness, examine CPU mode rates and load averages, check memory available and swap activity, evaluate disk space/inode and device latency, review network bandwidth, errors, and TCP retransmits, and finally correlate with application‑level metrics and logs. Any root‑cause conclusion must be supported by at least two independent evidence sources.

Acceptance testing and long‑term maintenance

For a new host class, record exporter version, enabled collectors, kernel, CPU core count, filesystem types, device mappings, NIC types, and scrape interval. Acceptance criteria include reachable /metrics, up=1, stable per‑core CPU rate (~1 second/second), memory totals matching /proc/meminfo, filesystem sizes matching df, correct device‑to‑storage mapping, and successful promtool validation of rules.

Never expose Node Exporter to untrusted networks; restrict listening address, firewall rules, and ACLs. If TLS or authentication is needed, configure the exporter’s web options (available in recent versions) or place it behind a controlled reverse proxy. All changes must be rolled out with gray‑deployment, verification of up, sample counts, and scrape latency, and a rollback plan that restores previous unit files and reloads Prometheus.

The ultimate goal is not to collect the most metrics but to ensure each alert answers three questions: Is the anomaly real? Where does it impact? What evidence should be gathered next? By aligning CPU, memory, disk, and network curves on a common timeline with configuration, logs, and business metrics, monitoring becomes actionable operational intelligence.