Why Switch to cgroup v2? Benefits and Step‑by‑Step Setup for Kubernetes

Linux has two cgroup versions: cgroup v1 and the newer cgroup v2. Starting with Kubernetes v1.25, the cgroup v2 feature is officially stable.

What are the advantages of cgroup v2

cgroup v2 provides a unified control system with enhanced resource management capabilities. Improvements over cgroup v1 include: Single unified hierarchy design in the API Safer subtree delegation to containers New features such as Pressure Stall Information (PSI) Enhanced resource allocation, management, and isolation across multiple resources Unified accounting of different memory types (network memory, kernel memory, etc.) Consideration of non‑instantaneous resource changes, e.g., page cache write‑back

Some Kubernetes features specifically use cgroup v2 to improve resource management and isolation, such as the MemoryQoS feature.

Prerequisites for using cgroup v2

cgroup v2 requires: Operating system distribution with cgroup v2 enabled Ubuntu (21.10+, recommended 22.04+) Debian GNU/Linux (11 Bullseye+) Fedora (31+) RHEL and RHEL‑like distributions (9+) Linux kernel 5.8 or newer Container runtime support for cgroup v2, e.g. containerd v1.4+, cri‑o v1.20+ Kubelet and container runtime configured to use the systemd cgroup driver

Using cgroup v2

Example using Debian 11 Bullseye + containerd v1.4.

Enable and check cgroup v2 on a Linux node

Debian 11 Bullseye enables cgroup v2 by default. Verify with:

stat -fc %T /sys/fs/cgroup/

Output cgroup2fs indicates cgroup v2; tmpfs indicates cgroup v1.

If not enabled, add systemd.unified_cgroup_hierarchy=1 to GRUB_CMDLINE_LINUX in /etc/default/grub and run sudo update-grub.

📝 Note: On Raspberry Pi, the standard OS does not enable cgroups by default. Append cgroup_memory=1 cgroup_enable=memory systemd.unified_cgroup_hierarchy=1 to /boot/cmdline.txt and reboot.

kubelet uses the systemd cgroup driver

kubeadm can pass a KubeletConfiguration struct during kubeadm init. The cgroupDriver field controls the kubelet’s cgroup driver.

In version 1.22, if the field is not set, kubeadm defaults it to systemd.

Minimal example configuring the field:

# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.21.0
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

Pass the file to kubeadm: kubeadm init --config kubeadm-config.yaml kubeadm writes the KubeletConfiguration to a ConfigMap in the kube-system namespace, which is then stored at /var/lib/kubelet/config.yaml on each node.

containerd uses the systemd cgroup driver

Edit /etc/containerd/config.toml:

[plugins.cri.containerd.runtimes.runc.options]
    SystemdCgroup = true

Upgrade monitoring components to support cgroup v2

Reference: Applications that directly access the cgroup filesystem must be updated to versions that support cgroup v2. Third‑party monitoring and security agents may need updates to support cgroup v2. If using cAdvisor as a DaemonSet, upgrade to v0.43.0 or newer. For Java workloads, use JDK 11.0.16+ or JDK 15+ for full cgroup v2 support.

Done 🎉🎉🎉

Summary

Kubernetes v1.25+ has cgroup v2 stable. Compared with cgroup v1, cgroup v2 offers:

Single unified hierarchy design in the API Safer subtree delegation to containers New features such as Pressure Stall Information (PSI) Enhanced resource allocation, management, and isolation across multiple resources Unified accounting of different memory types (network, kernel, etc.) Consideration of non‑instantaneous resource changes like page‑cache write‑back

It is recommended to use Linux distributions and container runtimes that support cgroup v2 when running Kubernetes v1.25 or later, and to enable the cgroup v2 feature in Kubernetes.

Link: https://url.hi-linux.com/pr6Yz

(© Original author, remove if requested)