Understanding kubectl top: How Kubernetes Metrics Are Collected and Interpreted

1. Introduction

kubectl top provides an easy way to view real‑time CPU and memory usage of nodes and pods. This article introduces its data flow and implementation principles, uses it to illustrate the Kubernetes monitoring system, and explains common issues.

Why does kubectl top report errors?

How does kubectl top node calculate values and how does it differ from the native top command?

How does kubectl top pod calculate values, does it include the pause container?

Why does kubectl top pod differ from the top command inside the pod?

Why do values from kubectl top pod differ from docker stats?

Tested on Kubernetes 1.8 and 1.13.

2. Usage

kubectl top is a basic command but requires the appropriate components to be deployed.

For versions < 1.8: deploy heapster.

For versions ≥ 1.8: deploy metrics‑server.

kubectl top node shows node usage:

kubectl top pod shows pod usage:

When no pod name is specified, all pods in the namespace are displayed; --containers shows all containers inside a pod.

Metric meanings:

CPU units are 100m = 0.1 core; memory units are 1Mi = 1024Ki.

Pod memory is the sum of all business containers, excluding the pause container, derived from container_memory_working_set_bytes.

Node values are not the sum of all pod values and differ from the values shown by the native top command.

3. Implementation Details

3.1 Data Flow

The data used by kubectl top, the Kubernetes dashboard, and HPA comes from the same source.

When using heapster, the apiserver proxies metric requests directly to the heapster service.

With metrics‑server, the apiserver accesses metrics via the /apis/metrics.k8s.io/ endpoint.

Example log comparison:

3.2 Metric API

Proxy forwarding is unstable and version‑uncontrolled. Heapster lacks full authentication and client integration. Therefore, metrics should be exposed as a first‑class API resource (Metric API).

Since version 1.8, heapster has been deprecated in favor of the Metric API implemented by metrics‑server.

3.3 kube‑aggregator

kube‑aggregator extends the apiserver, allowing custom services like metrics‑server to be registered under /apis/metrics.k8s.io. It provides dynamic registration, discovery, aggregation, and secure proxying.

3.4 Monitoring System

Metrics are divided into two categories:

Core metrics: collected from Kubelet/cAdvisor and provided by metrics‑server for Dashboard and HPA.

Custom metrics: provided via Prometheus Adapter (custom.metrics.k8s.io) for arbitrary Prometheus‑collected metrics.

Core metrics are sufficient for HPA, while custom metrics enable scaling based on application‑specific signals such as request QPS or error rates.

3.5 kubelet

Kubelet exposes two endpoints: 127.0.0.1:10255/metrics for summary metrics and 127.0.0.1:10255/metrics/cadvisor for container‑level metrics.

3.6 cAdvisor

cAdvisor, written in Go, collects container statistics (CPU, memory, network, filesystem) and provides an HTTP API. In Kubernetes it runs inside kubelet.

3.7 cgroup

cgroup files are the ultimate source of monitoring data, e.g., memory usage from

/sys/fs/cgroup/memory/docker/[containerId]/memory.usage_in_bytes

and limits from /sys/fs/cgroup/memory/docker/[id]/memory.limit_in_bytes. Memory usage ratio is calculated as memory.usage_in_bytes / memory.limit_in_bytes.

4. Common Issues

Missing heapster or metrics‑server, or pod crashes – check pod logs.

Pod just created – metrics may not be available for up to one minute.

Check if kubelet’s read‑only port (10255) is open; use the secure port (10250) if necessary.

4.2 Memory calculation for pods (including pause container)

Pause containers consume a small amount of memory (2‑3 Mi) but are excluded from the pod memory reported by kubectl top because cAdvisor does not list them. The reported memory is container_memory_working_set_bytes, calculated as:

container_memory_usage_bytes = container_memory_rss + container_memory_cache + kernel memory

container_memory_working_set_bytes = container_memory_usage_bytes – total_inactive_file

4.3 Node calculation vs native top

kubectl top node aggregates cgroup root statistics, not the sum of all pod metrics, and differs from the values shown by the native top command.

4.4 Difference between kubectl top pod and exec‑inside‑pod top

Top inside a pod shows the host’s total resources, not the pod’s allocated limits; cgroup RSS excludes shared memory and file cache.

4.5 Difference between kubectl top pod and docker stats

docker stats reports container_memory_usage_bytes - container_memory_cache, which differs from the values shown by kubectl top.

5. Conclusion

In most cases you do not need to monitor node or pod usage manually because Cluster Autoscaler and HPA handle scaling. Persisting cAdvisor data with Prometheus is recommended for historical analysis and alerts. Note that kubectl top help incorrectly lists support for storage up to version 1.16, and older versions require heapster while newer versions require metrics‑server.