Container Monitoring: Challenges, Metrics Collection, and Best Practices

Containers introduce a new monitoring dimension that traditional host‑oriented tools cannot fully capture, leading to blind spots and operational complexity.

Key challenges include the rapid lifecycle of containers, the risk of false host failures, and the need to avoid monitoring black holes between host and application layers.

Three metric groups are recommended: container‑level metrics, application metrics, and host metrics, each requiring specific collection methods.

Metrics collection methods :

Read pseudo‑files in /sys/fs/cgroup (e.g., /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpuacct.stat ) for CPU usage and throttling.

Inspect memory usage via files like /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.usage_in_bytes .

Gather I/O statistics from /sys/fs/cgroup/blkio/docker/$CONTAINER_ID/blkio.io_service_bytes and related files.

Obtain network counters by reading /proc/$CONTAINER_PID/net/dev after retrieving the container PID with docker inspect -f '{{ .State.Pid }}' $CONTAINER_ID .

Example command snippets:

CONTAINER_ID=$(docker run [OPTIONS] IMAGE [COMMAND] [ARG...])

# cat $CONTAINER_ID/cpuacct.stat
user 46409
system 22162

# cat $CONTAINER_ID/cpuacct.usage_percpu
362316789800
360108180815

The Docker CLI docker stats provides live per‑container CPU, memory, I/O, and network usage, while the Docker API (accessed via unix:///var/run/docker.sock ) offers richer detail for custom collectors.

Monitoring architectures :

Host‑centric monitoring treats containers as mini‑hosts but suffers from short lifetimes and false alarms.

A layered approach keeps host and application monitoring unchanged and adds a dedicated container layer, improving accuracy and reducing noise.

Alerting strategy focuses on internal network traffic changes to trigger alerts without flooding, using other resource metrics for root‑cause analysis.

Practical implementation at 数人云 combines cAdvisor, Prometheus, and Grafana, with custom agents for metric collection and aggregation, supporting high‑resolution data and service‑level visibility.

Q&A highlights cover Docker monitoring scope, toolchains (cAdvisor, Prometheus, Grafana), handling short container lifecycles, storage considerations, and security monitoring approaches.