How to Monitor Linux Conntrack Tables with Node‑Exporter’s Conntrack Plugin

Introduction

The node‑exporter conntrack plugin provides visibility into the Linux kernel's connection‑tracking table, which is crucial for firewalls, NAT gateways, and similar scenarios. A full conntrack table can block new connections, as illustrated by a production incident where nf_conntrack: table full caused service disruption.

Metrics Collected by the Plugin

When the firewall (e.g., systemctl start firewalld) is active, the plugin exposes a set of Prometheus metrics such as: node_nf_conntrack_entries – current allocated flow entries node_nf_conntrack_entries_limit – maximum table size node_nf_conntrack_stat_drop – packets dropped due to conntrack failure node_nf_conntrack_stat_early_drop – entries dropped to make room for new ones node_nf_conntrack_stat_found – successful lookups node_nf_conntrack_stat_invalid – packets that could not be tracked node_nf_conntrack_stat_insert – entries inserted node_nf_conntrack_stat_insert_failed – failed insert attempts node_nf_conntrack_stat_search_restart – lookups restarted after hash‑table resize

These can be queried with a simple curl, e.g. curl -s localhost:9100/metrics | grep "node_nf_conntrack_".

What Is Conntrack?

Conntrack is a Linux kernel module that tracks the state of network connections. In NAT scenarios it maps internal IP/port pairs to external ones, enabling the kernel to forward returning packets correctly. The table has a finite size; when it fills, new connections fail with the nf_conntrack: table full error. The current table can be inspected with conntrack -L.

Common Alert Rule

A typical alert checks the usage percentage:

100 * node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 85

If the usage exceeds 85 %, operators should consider increasing the table size, adjusting conntrack timeouts, or bypassing conntrack for certain traffic.

Collector Implementation Details

The Linux‑specific collector lives in conntrack_linux.go and follows the same pattern as other node‑exporter collectors, exposing init, NewConntrackCollector, and Update functions. The core logic resides in Update:

func (c *conntrackCollector) Update(ch chan<- prometheus.Metric) error {
    value, err := readUintFromFile(procFilePath("sys/net/netfilter/nf_conntrack_count"))
    if err != nil { return c.handleErr(err) }
    ch <- prometheus.MustNewConstMetric(c.current, prometheus.GaugeValue, float64(value))

    value, err = readUintFromFile(procFilePath("sys/net/netfilter/nf_conntrack_max"))
    if err != nil { return c.handleErr(err) }
    ch <- prometheus.MustNewConstMetric(c.limit, prometheus.GaugeValue, float64(value))

    conntrackStats, err := getConntrackStatistics()
    if err != nil { return c.handleErr(err) }

    ch <- prometheus.MustNewConstMetric(c.found, prometheus.GaugeValue, float64(conntrackStats.found))
    ch <- prometheus.MustNewConstMetric(c.invalid, prometheus.GaugeValue, float64(conntrackStats.invalid))
    ch <- prometheus.MustNewConstMetric(c.ignore, prometheus.GaugeValue, float64(conntrackStats.ignore))
    ch <- prometheus.MustNewConstMetric(c.insert, prometheus.GaugeValue, float64(conntrackStats.insert))
    ch <- prometheus.MustNewConstMetric(c.insertFailed, prometheus.GaugeValue, float64(conntrackStats.insertFailed))
    ch <- prometheus.MustNewConstMetric(c.drop, prometheus.GaugeValue, float64(conntrackStats.drop))
    ch <- prometheus.MustNewConstMetric(c.earlyDrop, prometheus.GaugeValue, float64(conntrackStats.earlyDrop))
    ch <- prometheus.MustNewConstMetric(c.searchRestart, prometheus.GaugeValue, float64(conntrackStats.searchRestart))
    return nil
}

The collector reads two proc files: /proc/sys/net/netfilter/nf_conntrack_count – current entry count /proc/sys/net/netfilter/nf_conntrack_max – maximum allowed entries

It then calls getConntrackStatistics, which parses /proc/net/stat/nf_conntrack. This file contains hexadecimal counters for fields such as entries, found, invalid, ignore, insert, insert_failed, drop, early_drop, and search_restart. The exporter converts these values to decimal and aggregates them across multiple lines.

Field Definitions (Excerpt from man‑page)

entries          Number of entries in conntrack table.
searched         Number of conntrack table lookups performed.
found            Number of searched entries that were successful.
new              Number of conntrack entries added that were not expected before.
invalid          Number of packets that cannot be tracked.
ignore           Number of packets already associated with a conntrack entry.
delete           Number of conntrack entries removed.
delete_list      Number of entries moved to the dying list.
insert           Number of entries inserted into the list.
insert_failed    Number of insertion attempts that failed (e.g., duplicate).
drop             Number of packets dropped due to conntrack failure.
early_drop      Number of entries dropped to make room for new ones.
search_restart   Number of lookups restarted because the hash table was resized.

Conclusion

The conntrack plugin for node‑exporter is a valuable tool for monitoring the health of Linux connection‑tracking tables in firewall or NAT deployments. By exposing detailed counters and providing a simple alert rule, operators can proactively detect table saturation and take corrective actions before connections are disrupted.