Mastering Kubernetes Garbage Collection: Tips, Parameters, and Real-World Experiments

Tips

Kubernetes garbage collection is managed by kubelet, which checks containers every minute and images every five minutes; the first container GC occurs one minute after kubelet start, and the first image GC occurs five minutes after start.

Do not use other tools or manual cleanup for containers and images because kubelet determines pod status from containers; external removal may disrupt kubelet.

Image reclamation applies to all images managed by Docker on a node, regardless of whether they were pulled for a pod; container reclamation only applies to containers managed by kubelet.

Kubernetes uses cAdvisor integrated in kubelet for image reclamation with two parameters: --image-gc-high-threshold and --image-gc-low-threshold . When disk usage reaches the high threshold, image GC is triggered and removes least‑recently‑used images until usage falls below the low threshold. Defaults are 90% (high) and 80% (low).

Container reclamation has three configurable parameters: --minimum-container-ttl-duration , --maximum-dead-containers-per-container , and --maximum-dead-containers . After a stopped container has existed for the TTL duration it is marked expired (not yet removed). By default the TTL is 1m0s, each pod may keep up to 2 dead containers, and a node may keep up to 100 dead containers.

To disable container GC, set --minimum-container-ttl-duration to 0 and set the other two parameters to negative values.

The TTL value can use unit suffixes, e.g., h for hours, m for minutes, s for seconds.

If --maximum-dead-containers-per-container and --maximum-dead-containers conflict, the node‑wide --maximum-dead-containers takes precedence.

Containers created by kubelet without a name are deleted when they reach their GC point.

When reclaiming containers, they are sorted by creation time and the oldest are removed first.

At a GC point, kubelet performs the following steps: (1) iterate all pods to satisfy --maximum-dead-containers-per-container ; (2) if still exceeding --maximum-dead-containers , compute X = (max‑dead‑containers) / (total pods) and ensure each pod has at most X dead containers (minimum 1); (3) if still exceeding, delete the oldest dead containers until the limit is met.

When an image is pulled again or used by a new pod, its last‑used timestamp is refreshed.

Kubernetes GC became functional starting with version 1.1.4; earlier versions contain bugs and may not work.

Pod identity matters for container reclamation: deleting and recreating a pod with the same name does not treat the old pod’s containers as part of the new pod’s dead‑container set.

Experiments

Image reclamation using Docker default --graph ( /var/lib/docker ) with kubelet flags --image-gc-high-threshold=40 and --image-gc-low-threshold=35 . After creating two pods with different images, disk usage rose to 41% and the image centos:7 was not removed until the associated containers were manually deleted with docker rm , after which the image was reclaimed.

Image reclamation with a custom Docker graph directory ( /opt ) and thresholds --image-gc-high-threshold=50 , --image-gc-low-threshold=40 . Even after deleting the pod and manually removing containers, the image remained, indicating a bug when --graph points to a non‑default location. The issue is tracked at https://github.com/kubernetes/kubernetes/issues/17994 and requires adding --docker-root=<graph‑path> to kubelet.

Container reclamation using --maximum-dead-containers-per-container=1 , --minimum-container-ttl-duration=30s , and --maximum-dead-containers=100 . A pod with a single short‑lived container repeatedly exited; after the TTL, only one dead container per pod remained, and the oldest were removed first.

Testing whether --maximum-dead-containers-per-container counts containers or container sets. With three containers in a pod, after GC only three containers (one per container set) remained, confirming the parameter counts per pod‑level container set.

Container reclamation using --maximum-dead-containers=3 (with --maximum-dead-containers-per-container=2 and TTL 30s). After creating and deleting a pod multiple times, eight dead containers existed; after GC only three remained, and pause containers were also reclaimed.

Checking if non‑kubelet‑managed containers are counted by --maximum-dead-containers . A manually created container persisted after GC, demonstrating that the GC strategy does not apply to containers outside kubelet management.

Conclusion: Only when all containers associated with an image are stopped and removed can Kubernetes image garbage collection reclaim the image.

Conclusion: The --maximum-dead-containers-per-container parameter works as expected for container GC.

Conclusion: The --maximum-dead-containers-per-container count applies to a pod’s container set, not to individual containers.

Conclusion: The --maximum-dead-containers parameter works as expected; pause containers are also managed and reclaimed.

Conclusion: Kubernetes container GC does not apply to containers not managed by kubelet.