How to Build a Container Monitoring Stack with CAdvisor, InfluxDB, and Grafana

1 Container Monitoring Solution Selection

When evaluating container monitoring tools we considered many options such as the built‑in docker stats, Scout, DataDog, Sysdig Cloud, Sensu Monitoring Framework, and CAdvisor. docker stats provides real‑time CPU, memory and network data but lacks persistence and alerting. Hosted services like Scout, DataDog and Sysdig Cloud are feature‑rich but are paid SaaS solutions, which we excluded. Sensu is free but complex to deploy. We finally chose CAdvisor because it is open‑source, provides a full set of metrics, and has an official Docker image.

CAdvisor, developed by Google, is easy to deploy and can be combined with InfluxDB for storage and Grafana for visualization, resulting in a lightweight yet powerful monitoring stack.

2 Container Resource Monitoring – CAdvisor

2.1 Deployment and Operation

CAdvisor runs as a Docker container and can be started with the following command:

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

After the container starts, the UI is available at http://<em>host_ip</em>:8080.

2.2 Integrating InfluxDB

Because CAdvisor only keeps two minutes of data locally, we configure it to write metrics to an InfluxDB instance. The container definition includes the InfluxDB connection details:

{
    "binds": [
        "/:/rootfs:ro",
        "/var/run:/var/run:rw",
        "/sys:/sys:ro",
        "/home/docker/var/lib/docker/:/var/lib/docker:ro"
    ],
    "image": "forum-cadvisor",
    "labels": {"type": "cadvisor"},
    "command": "-docker_only=true -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=influxdb.service.consul:8086 -storage_driver_user=testuser -storage_driver_password=testpwd",
    "tag": "latest",
    "hostname": "cadvisor-${lan_ip}"
}

We use a custom image forum-cadvisor to include necessary fixes and utilities.

2.3 Known Issues and Fixes

1) Runtime error – missing findutils

The container logs show errors because the find command lacks the -printf option. Installing findutils resolves the issue.

2) Missing memory metrics

Debian kernels may disable CGroup memory support. Adding cgroup_enable=memory to /etc/default/grub and updating GRUB fixes the problem: GRUB_CMDLINE_LINUX="cgroup_enable=memory" Then run sudo update-grub2 && reboot.

3) Incorrect network traffic data

CAdvisor originally reports only the first network interface. We patched the source to aggregate all interfaces and rebuilt the binary, then placed it in a custom Dockerfile:

FROM google/cadvisor:latest
RUN apk add --update findutils && rm -rf /var/cache/apk/*
COPY src/cadvisor /usr/bin/cadvisor

2.4 How CAdvisor Works

CAdvisor mounts the host’s root filesystem and Docker directories, reads cgroup files under /sys/fs/cgroup/ for CPU, memory, and I/O statistics, and extracts network data from /proc/<em>PID</em>/net/dev. Example commands:

# cat /sys/fs/cgroup/cpu/docker/b1f25723c5c3a17df5026cb60e1d1e1600feb293911362328bd17f671802dd31/cpuacct.stat
user 95191
system 5028

# cat /proc/6748/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  eth0: 6266314 512 0 0 0 0 0 0 22787 292 0 0 0 0 0 0
  lo: 5926805 5601 0 0 0 0 0 0 5926805 5601 0 0 0 0 0 0

3 Container Monitoring Data Storage – InfluxDB

3.1 InfluxDB Configuration and Run

InfluxDB is a distributed time‑series database written in Go. We run it as a Docker container with the following configuration:

{
    "binds": ["${volume_dir}/influxdb/data:/var/lib/influxdb"],
    "environment": {
        "SERVICE_INFO": {
            "Name": "influxdb",
            "Address": "${register_ip}",
            "Port": 8086
        }
    },
    "image": "influxdb",
    "name": "influxdb-${namespace}",
    "tag": "latest"
}

After starting the container we create a database and user:

# influx
> create database cadvisor
> CREATE USER testuser WITH PASSWORD 'testpwd'
> GRANT ALL PRIVILEGES ON cadvisor TO testuser
> CREATE RETENTION POLICY "cadvisor_retention" ON "cadvisor" DURATION 30d REPLICATION 1 DEFAULT

InfluxDB automatically creates measurement tables for CAdvisor metrics (e.g., cpu_usage_total, memory_usage, rx_bytes, etc.).

3.2 Important InfluxDB Concepts

Database : logical container for measurements.

Timestamp : the time column in each point.

Fields : key‑value pairs without indexes (e.g., value).

Tags : indexed key‑value pairs (e.g., container_name, host).

Retention Policy : defines data lifespan (e.g., cadvisor_retention = 30 days).

Measurement : similar to a table, grouping fields, tags, and timestamps.

Series : a set of points sharing the same measurement, retention policy, and tag set.

Point : a single record in a series.

3.3 InfluxDB Features

Special functions such as FILL(), INTEGRAL(), SPREAD(), STDDEV(), MEAN(), MEDIAN(), SAMPLE(), and DERIVATIVE() help with analysis. Continuous queries enable periodic down‑sampling of data for long‑term storage.

4 Container Monitoring Visualization – Grafana

Grafana runs in its own container and connects directly to the InfluxDB service via the internal DNS name influxdb.service.consul:8086. Container configuration:

{
    "binds": [
        "${volume_dir}/grafana/data:/var/lib/grafana",
        "${volume_dir}/grafana/log:/var/log/grafana"
    ],
    "environment": {"GF_SECURITY_ADMIN_PASSWORD": "testpwd"},
    "image": "grafana/grafana",
    "name": "grafana-${namespace}",
    "port_bindings": {"3000": 8888},
    "ports": [3000],
    "tag": "latest"
}

After starting, the UI is reachable at http://<em>host_ip</em>:8888/. We configure InfluxDB as a data source and create panels to display metrics such as rx_bytes and memory_usage, selecting the data (IEC) unit for byte‑type values.

5 Summary

Using CAdvisor + InfluxDB + Grafana to build a container resource monitoring system is feasible and straightforward. All three components run as containers, fitting our fully‑containerized production environment. The system is now in production, providing reliable visual monitoring and supplying runtime data for anomaly detection and intelligent scheduling algorithms.