How to Decode Container CPU Metrics in Prometheus and Docker Stats

Parameter Explanation

When configuring Prometheus to monitor container CPU usage in Kubernetes, you may encounter CPU usage exceeding 100%. The following metrics are relevant:

container_spec_cpu_period : The CFS scheduling time window (CPU clock period) for a container, usually 100,000 µs.

container_spec_cpu_quota : Total CPU time allocated to the container; e.g., a quota of 700,000 µs corresponds to the value of resource.cpu.limits in Kubernetes.

container_spec_cpu_share : Relative share of host CPU requested, e.g., 500m means 0.5 CPU (50,000 µs), matching resource.cpu.requests.

container_cpu_usage_seconds_total : Cumulative CPU usage of the container in seconds, summed across all cores.

container_cpu_system_seconds_total : CPU time spent in kernel mode.

container_cpu_user_seconds_total : CPU time spent in user mode.

Reference: https://docs.signalfx.com/en/latest/integrations/agent/monitors/cadvisor.html https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md

Specific Formulas

Default query using container_cpu_usage_seconds_total:

sum(irate(container_cpu_usage_seconds_total{container="$Container",instance="$Node",pod="$Pod"}[5m])*100) by (pod)

This returns the average usage across all cores of the container.

To calculate per‑container CPU usage as a percentage:

sum(irate(container_cpu_usage_seconds_total{container="$Container",instance="$Node",pod="$Pod"}[5m])*100) by (pod) /
sum(container_spec_cpu_quota{container="$Container",instance="$Node",pod="$Pod"} / container_spec_cpu_period{container="$Container",instance="$Node",pod="$Pod"}) by (pod)

Here container_spec_cpu_quota/container_spec_cpu_period represents the number of cores allocated to the container.

Official GitHub issue: https://github.com/google/cadvisor/issues/2026#issuecomment-415819667

Docker Stats

The docker stats command obtains live data from the Docker API /containers/(id)/stats. Memory usage shown excludes cache; the field memory_stats.total_inactive_file (Docker ≤19.03) or memory_stats.cache (Docker >19.03) holds cache information. The PIDS column counts processes/threads created by the container (lightweight processes & kernel tasks).

Example curl command to fetch raw stats:

curl -s --unix-socket /var/run/docker.sock "http://localhost/v1.40/containers/10f2db238edc/stats" | jq -r ...

Key fields used for calculations:

used_memory = memory_stats.usage - memory_stats.stats.cache

available_memory = memory_stats.limit

Memory usage % = (used_memory / available_memory) * 100

cpu_delta = cpu_stats.cpu_usage.total_usage - precpu_stats.cpu_usage.total_usage

system_cpu_delta = cpu_stats.system_cpu_usage - precpu_stats.system_cpu_usage

number_cpus = length(cpu_stats.cpu_usage.percpu_usage) or cpu_stats.online_cpus

CPU usage % = (cpu_delta / system_cpu_delta) * number_cpus * 100

Docker API reference: https://docs.docker.com/engine/api/v1.40/#operation/ContainerStats