Using Pod Overhead and Kata Containers to Isolate Kernel Memory and Stop Container Slab Leaks

Background

In containerized workloads, processing malformed audio files can trigger severe memory leaks. The leak manifests as continuous growth of kernel slab memory that cannot be reclaimed, eventually exhausting host memory, causing OOM or host unresponsiveness, and threatening multi‑tenant stability.

Reproducing the Issue

A pod is created to simulate massive file reads/writes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: slabtest-kata
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: slabtest-kata
  template:
    metadata:
      labels:
        app: slabtest-kata
    spec:
      nodeName: p92330v.hulk.bjmd.qihoo.net
      containers:
      - name: slabtest-kata
        image: harborqa.qihoo.net/360cloud/slabtest:latest
        imagePullPolicy: IfNotPresent
        command: ["/usr/local/bin/slab_dentry_leak"]
        args: ["-base", "/slabtest", "-dirs", "2000", "-files", "20000", "-mode=positive","-loop=true","-fsize=4096"]
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "2Gi"]

Diagnosis

Host‑level inspection steps:

top : monitor rapid decrease of free memory and increase of buff/cache.

meminfo : watch Slab and SReclaimable growth using

watch -n1 "grep -E 'MemTotal|MemFree|MemAvailable|Cached|Slab|SReclaimable|SUnreclaim' /proc/meminfo"

.

slabinfo : filter dentry/inode related entries with cat /proc/slabinfo | egrep -i 'dentry|inode|path|ext4|xfs'.

pidstat : identify the process performing heavy I/O via pidstat -d 1.

Map the identified PID to a container ID (docker or containerd) and then to a sandbox ID, as shown in the images.

Root Cause

Analysis of top and memory metrics shows that continuous heavy file‑system operations inflate VFS metadata caches (inode, dentry, etc.), driving SReclaimable far above normal while SUnreclaim remains low, indicating the leak is not due to unrecoverable kernel objects but to excessive cache growth.

Why Ordinary Containers Cannot Contain the Leak

Standard containers use the runC runtime with cgroup paths such as:

/sys/fs/cgroup/memory/kubepods

/sys/fs/cgroup/memory/kubepods/burstable/

/sys/fs/cgroup/memory/kubepods/burstable/pod_id

/sys/fs/cgroup/memory/kubepods/burstable/pod_id/sandbox_id

Even though the pod’s resources.limits.memory is reflected in memory.limit_in_bytes of the container cgroup, file I/O still occurs on the host kernel, so the cgroup cannot restrict the kernel‑level slab growth.

# View pod cgroup limits
cd /sys/fs/cgroup/memory/kubepods/
cat memory.limit_in_bytes memory.max_usage_in_bytes memory.usage_in_bytes
4294967296=2Gi+2Gi
12455936
5042176

Kata Containers and Sandbox Overhead

Kata introduces a sandbox VM; its shim and VMM processes consume additional host resources. When sandbox_cgroup_only=false, the sandbox runs in a dedicated /kata_overhead cgroup that has no limits, so memory consumption is unchecked.

# Only pod‑level cgroup
cd /sys/fs/cgroup/memory/kubepods/
cat memory.limit_in_bytes memory.max_usage_in_bytes memory.usage_in_bytes
6442450944 / 0 / 0

# kata_overhead cgroup
cd /sys/fs/cgroup/memory/kata_overhead/
cat memory.limit_in_bytes memory.max_usage_in_bytes memory.usage_in_bytes
9223372036854771712 / 251486208 / 251043840

When sandbox_cgroup_only=true, Kubernetes adds the sandbox’s resource needs to the pod’s overhead via PodOverhead. The pod cgroup then enforces limits on both container and sandbox processes.

# pod‑level cgroup with overhead
cd /sys/fs/cgroup/memory/kubepods/
cat memory.limit_in_bytes memory.max_usage_in_bytes memory.usage_in_bytes
6547308544=1Gi+2Gi+3Gi+100Mi / 242954240 / 242388992

Pod Overhead Configuration

The RuntimeClass is defined with a fixed overhead that reserves CPU and memory for the Kata sandbox:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-qemu
handler: kata-qemu
scheduling:
  nodeSelector:
    katacontainers.io/kata-runtime: "true"
overage:
  podFixed:
    cpu: "100m"
    memory: "100Mi"

Conclusion

File‑system I/O in a pod expands kernel slab memory regardless of user‑space memory limits. Ordinary containers cannot isolate this kernel memory because cgroup limits apply only to user‑space processes. Kata containers separate the workload into a sandbox VM; by configuring PodOverhead and enabling sandbox_cgroup_only=true, the sandbox’s memory consumption—including slab growth—is accounted for in the pod’s cgroup limits, causing the sandbox to be OOM‑killed before the host is affected.