Skip to content
Why OOM Happens Despite Free Memory: Linux cgroups, Kubernetes & systemd-oomd
Operations 31 min read

Why OOM Happens Despite Free Memory: Linux cgroups, Kubernetes & systemd-oomd

This article explains why Linux servers trigger OOM kills even when free memory appears available, covering global OOM, cgroup limits, Kubernetes OOMKilled, systemd-oomd, NUMA constraints, and application-level memory errors, with a step-by-step evidence-based troubleshooting methodology.

Raymond Ops
Raymond Ops
Raymond Ops

1. Identify Which Type of OOM Occurred

Do not assume every process exit is an OOM. Distinguish at least these cases:

Global OOM : Host-wide memory reclaim fails; kernel OOM killer selects and terminates a process.

cgroup OOM : A service or container hits its own memory limit, triggering OOM inside that cgroup even if the host has plenty of free memory.

Kubernetes Container OOMKilled : Container process exceeds its memory limit or its cgroup suffers OOM; kubelet/runtime records exit reason as OOMKilled.

systemd-oomd Proactive Kill : systemd-oomd uses PSI (Pressure Stall Information) to choose a cgroup and send a termination signal. It is a user-space policy daemon, not the kernel OOM killer.

Application-Level Memory Error : e.g., JVM throws java.lang.OutOfMemoryError, database internal memory limit errors, malloc failures. These do not necessarily involve kernel process kills.

External Signal Termination : Deployment systems, probes, admins, or process managers may send signals. Exit code 137 usually means SIGKILL, but SIGKILL sources are not only OOM.

The first goal is to confirm the terminator and the limiting boundary, not to immediately tune parameters.

2. Collect On-Site Evidence: Preserve Proof

Record fault time, host, process/Pod, app version, recent changes, and monitoring sampling interval. Confirm current node and system time:

hostnamectl
date --iso-8601=seconds
uptime

Immediately save memory overview:

free -h
cat /proc/meminfo
vmstat 1 10
ps -eo pid,ppid,user,stat,rss,vsz,pmem,etime,comm,args --sort=-rss | head -n 30
ps

RSS and VSZ are only for initial screening. Shared pages cause double-counting; virtual address space ≠ physical memory used. To analyze a single process's actual mappings, read:

PID=12345
cat "/proc/$PID/status"
cat "/proc/$PID/smaps_rollup"
cat "/proc/$PID/limits"
PID=12345

is a placeholder; replace with the target PID still alive. If the process has exited, /proc/$PID disappears; turn to kernel logs, service logs, monitoring, and cgroup counters.

If writing output to files, check diagnostic directory capacity and inodes first. Do not delete system logs because of disk pressure: OOM logs are root-cause evidence; deleting or truncating breaks analysis and audit.

df -h /tmp
df -i /tmp

3. Correctly Interpret free : free ≠ available

Run: free -h Example output:

total        used        free      shared  buff/cache   available
Mem:           31Gi        24Gi       600Mi        1.2Gi        6.4Gi        5.3Gi
Swap:         4.0Gi        1.1Gi       2.9Gi

Low free is usually not a problem. Linux uses free memory for page cache and reclaims when needed. More valuable than free is MemAvailable: kernel's estimate of memory available for new applications without swapping, based on free pages, reclaimable page cache, watermarks, etc. — but still an estimate.

Situations where "available is high now but OOM happened before":

OOM killer already killed a large process; memory released on exit.

Monitoring samples every 1 minute; memory spike occurred and ended between two samples.

It was a cgroup OOM; host MemAvailable is not that cgroup's available quota.

Monitoring shows node total memory, but app is constrained by container limit, systemd MemoryMax=, or other hierarchical limits.

Failed allocation required a specific memory zone or contiguous pages; host-summarized available cannot guarantee that allocation succeeds.

Therefore, do not use post-fault free -h to disprove OOM logs, nor rely on a single total-memory curve to judge limits.

4. Check Kernel Logs First: Confirm Global OOM or memcg OOM

On systemd systems, query the fault window:

journalctl -k --since '2026-07-10 14:00:00' --until '2026-07-10 14:30:00' --no-pager

Replace times with actual fault window. Filter by common keywords:

journalctl -k --since '-2 hours' --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

Systems without persistent journal: check /var/log/messages, /var/log/syslog, or log platform; location depends on distro and rsyslog/journald config. Container dmesg may not see host logs due to permissions/namespaces.

Kernel OOM records typically include triggering task, gfp_mask, order, constraint info, process memory list, and final Killed process. Fields vary by kernel version; don't rely on a single fixed regex. Focus on:

Does log explicitly show oom-kill or Out of memory?

Do constraint, mems_allowed, nodemask indicate cpuset, NUMA, or memcg constraints?

Does Memory cgroup out of memory or a cgroup path appear?

Killed PID, process name, total-vm, anon-rss, file-rss, shmem-rss.

Memory reclaim, swap, kernel warnings, process starts before/after fault.

Example log shapes (fields not guaranteed across kernels):

kernel: oom-kill:constraint=CONSTRAINT_MEMCG,...,task=example,pid=12345,...
kernel: Memory cgroup out of memory: Killed process 12345 (example) ...

If log shows CONSTRAINT_MEMCG or explicit memory cgroup path, host having available memory is expected: failure occurred inside cgroup limit. If global OOM, combine with /proc/meminfo history, swap activity, slab, page tables, and kernel logs to determine why reclaim failed.

5. Examine cgroup: Most Common "Host Has Memory, App Still OOM"

First find which cgroup the target process belongs to: cat "/proc/$PID/cgroup" cgroup v2 output usually contains unified hierarchy path. Build the directory from the actual path; placeholder example:

CGROUP_PATH=/sys/fs/cgroup/system.slice/example.service
cat "$CGROUP_PATH/memory.current"
cat "$CGROUP_PATH/memory.max"
cat "$CGROUP_PATH/memory.high"
cat "$CGROUP_PATH/memory.events"
cat "$CGROUP_PATH/memory.stat"
cat "$CGROUP_PATH/memory.pressure"

Replace CGROUP_PATH with the real directory from /proc/$PID/cgroup. Key fields: memory.current: current memory usage of cgroup and descendants. memory.max: hard limit; value max means no hard limit set. memory.high: high watermark; exceeding triggers reclaim and throttling, but not a hard limit. memory.events: cumulative counters like low, high, max, oom, oom_kill (fields depend on kernel version). memory.stat: breakdown of anon, file, kernel, slab, etc. memory.pressure: PSI memory pressure info. memory.events are cumulative; seeing oom_kill 3 only means 3 total in cgroup lifetime, not that another just happened. Compare pre/post fault deltas or correlate with log timestamps.

Systemd service limits can also be confirmed via:

systemctl show example.service -p MemoryCurrent -p MemoryHigh -p MemoryMax -p MemorySwapMax -p OOMPolicy
systemctl cat example.service

Replace example.service with real unit name. Check drop-in configs under /etc/systemd/system/example.service.d/*.conf which may override main unit.

Don't check only the service level. cgroup limits are hierarchical; parent slice's MemoryMax= may limit combined usage of multiple child services. Use:

systemd-cgls
systemd-cgtop

6. Confirm OOMKilled in Kubernetes

Kubernetes commands must specify namespace. Placeholders: $NAMESPACE, $POD, $CONTAINER:

NAMESPACE=production
POD=example-pod
CONTAINER=example-container
kubectl -n "$NAMESPACE" get pod "$POD" -o wide
kubectl -n "$NAMESPACE" describe pod "$POD"
kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"
"}{end}'
kubectl -n "$NAMESPACE" logs "$POD" -c "$CONTAINER" --previous --timestamps

If container never restarted or previous instance logs unavailable, --previous may error. Also check resource limits in workload manifest:

kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{range .spec.containers[*]}{.name}{"\trequests="}{.resources.requests.memory}{"\tlimits="}{.resources.limits.memory}{"
"}{end}'
kubectl -n "$NAMESPACE" get events --field-selector involvedObject.name="$POD" --sort-by='.metadata.creationTimestamp'

Events have retention; "no events" ≠ "no OOM". Complete evidence includes container lastState.terminated.reason, node kernel/runtime logs, cgroup memory.events delta, and pre-fault container memory curve.

Common Prometheus container metrics: container_memory_working_set_bytes, container_memory_rss, container_memory_cache, container_memory_failcnt — but collection chain and cgroup version affect availability and semantics; must note: rely on actual exporter-exposed metrics. working_set cannot be simply equated to "non-reclaimable memory".

If Pod has no memory limit, it can still be killed by node global OOM; Kubernetes QoS class and oom_score_adj influence selection under node pressure but are not absolute protection. Check:

kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{.status.qosClass}{"
"}'

7. systemd-oomd: Not Kernel OOM, Can Still Kill Services

Verify service status and logs:

systemctl status systemd-oomd --no-pager
journalctl -u systemd-oomd --since '-2 hours' --no-pager
oomctl
oomctl

availability depends on systemd version. systemd-oomd typically uses cgroup v2 and PSI for policy actions; related units can configure ManagedOOMSwap=, ManagedOOMMemoryPressure= etc.; exact support must be verified via local man systemd.resource-control and systemd version.

If logs clearly show systemd-oomd killed a cgroup, root cause cannot be written as "kernel memory exhausted". Continue to confirm: which parent cgroup enabled management policy, how long pressure lasted, swap usage, whether service memory growth matches business load.

Do not disable systemd-oomd just to avoid mis-kills. Disabling changes whole-host memory pressure protection, potentially turning controllable service terminations into system-wide hangs or global OOM. Any policy adjustment must first identify affected slice, validate pressure thresholds, and prepare rollback.

8. Separate Application OOM from Kernel OOM

JVM OutOfMemoryError may originate from heap, metaspace, direct buffers, or thread creation — not necessarily kernel kill. Even if Java heap hasn't hit container limit, native memory, thread stacks, JIT, and shared libraries count toward container memory. For still-running processes with JDK tools available:

jcmd "$PID" VM.flags
jcmd "$PID" GC.heap_info
jcmd "$PID" VM.native_memory summary

Native Memory Tracking must be enabled at JVM startup; otherwise last command gives incomplete info. Heap dumps cause pauses and heavy disk I/O; do not run when disk space is low or on primary instance during peak. Databases, Go, Node.js, Python have their own memory models; cannot apply JVM parameters. Use corresponding version's diagnostic interfaces and align with cgroup and kernel data. When app reports allocation failure but kernel has no OOM record, check app's own limits, runtime error types, and /proc/$PID/limits.

9. Global OOM Further Evidence: Where Did Memory Go?

From /proc/meminfo watch AnonPages, Shmem, SReclaimable, SUnreclaim, Slab, PageTables, HugePages_Free, Committed_AS, CommitLimit (field existence depends on kernel config). View slab: slabtop -o When SUnreclaim or a slab cache grows continuously, correlate with time series, kernel version, and workload; a single large value doesn't prove kernel leak. tmpfs also consumes memory:

findmnt -t tmpfs
df -h -t tmpfs

Deleted but still-open tmpfs files don't release immediately. lsof +L1 finds such files; limit scope on large hosts.

Check swap and PSI:

swapon --show
vmstat 1 10
cat /proc/pressure/memory
vmstat
si

/ so = pages swapped in/out per second; existing swap usage ≠ active swapping. PSI some and full reflect tasks stalled due to memory pressure; compare with historical baseline. No swap isn't necessarily wrong; swap doesn't solve continuous leaks; Kubernetes node swap support and policy have version differences.

Check overcommit:

sysctl vm.overcommit_memory vm.overcommit_ratio
grep -E 'CommitLimit|Committed_AS' /proc/meminfo
Committed_AS

is not current RSS. Changing overcommit alters allocation behavior for databases, caches, large forks; high-risk kernel change. NUMA hosts must also check allowed memory nodes for target process:

numactl --hardware
numastat
numastat -p "$PID"
cat "/proc/$PID/status" | grep -E 'Mems_allowed|Cpus_allowed'
numactl

/ numastat from numactl package. Combine with OOM log's mems_allowed and binding policy; don't modify NUMA or overcommit params without evaluating latency/bandwidth impact.

10. Derive Root Cause from Evidence, Not Guess from Result

Root cause must be supported by combined evidence from same time window: cgroup OOM needs memory.current / memory.max, memory.events delta, and kernel logs; memory leak needs sustained anon/runtime heap growth under same version/load plus object/allocation path evidence; transient spike needs high-res concurrency and memory curves; global OOM needs kernel logs, MemAvailable, swap, PSI, reclaim degradation; systemd-oomd action needs its logs and policy evidence.

Distinguish trigger condition from business root cause. memory.max hit is proximate cause; business root cause may be unbounded cache, increased batch concurrency, or limit mismatched to actual working set. High RSS, exit code 137, or "recovery after restart" alone cannot prove leak or kernel OOM.

11. Remediation: Control Risk First, Then Adjust Limits

1. Temporary Control

Prefer app's existing rate limiting, concurrency control, task pause, cache caps, or traffic draining. Pausing batch jobs or lowering concurrency may affect data freshness and backlog; confirm queue length, task idempotency, recovery speed, and business SLA before executing.

2. Adjust Container or systemd Memory Limits

Raising limit is not cost-free. Kubernetes Deployment resource changes usually trigger rolling update; systemd MemoryMax= change may need daemon-reload and service restart or property re-apply, depending on method.

High-risk operations must define:

Impact scope : target Pod/service/parent slice, and other workloads' available memory on node.

Pre-execution checks : current working set, peak, node allocatable memory, remaining capacity, QoS, PDB, replica count, rolling strategy.

Backup method : save original Kubernetes manifests, Helm values, systemd units and drop-ins; save OOM logs and metric snapshots.

Canary method : test in staging or single replica/node first, restrict traffic, observe at least one representative business cycle. Use version-controlled manifest diff; don't manually edit all nodes.

Validation : verify memory.events stops growing, working set/RSS bounded, PSI normal, error rate/latency no regression, node headroom sufficient.

Rollback plan : restore original manifest or previous release, re-roll; if limit increase causes node pressure, halt rollout and shift traffic back to healthy instances.

Kubernetes can do client-side dry-run for syntax validation, but dry-run cannot verify scheduling capacity or runtime memory behavior:

kubectl -n "$NAMESPACE" apply --dry-run=client -f workload.yaml
workload.yaml

is placeholder for actual manifest. Formal changes should also pass code review, server-side validation, and established release pipeline.

3. Add or Adjust Swap

Creating swap file, running mkswap / swapon, modifying /etc/fstab alter host memory behavior and boot config; high-risk. Impact scope: entire host. Pre-check: disk space, filesystem support, encryption requirements, Kubernetes/kubelet policy, app tolerance for swap latency. Backup /etc/fstab and current swapon --show output; validate on similar test node first, then roll out per node.

Validation must include swapon --show, free -h, vmstat, PSI, app P95/P99 latency, and node reboot mount status. Rollback requires confirming swap usage safely migrated; cannot swapoff when memory headroom insufficient, else may trigger OOM. This article does not provide copy-paste swap creation commands to avoid misuse before confirming filesystem, disk, and cluster policy.

4. Modify Kernel Parameters

vm.overcommit_memory

, vm.overcommit_ratio, vm.swappiness, HugePages, NUMA params can affect all workloads. Pre-change: record original values, confirm app/database official recommendations, stress-test on same-config nodes, roll out via config management. Temporary sysctl -w vs persistent config have different effect scopes; rollback must restore both runtime values and persistent files to avoid reversion on reboot.

12. Fix Verification & Rollback Criteria

After fix, don't just verify "process not killed again". At minimum observe: memory.current vs memory.max distance, and memory.events delta.

Node MemAvailable, swap activity, PSI, slab, page tables stability.

App working set, anon memory, heap, native memory reaching stable plateau.

Request volume, concurrency, error rate, throughput, P95/P99 latency recovery.

Kubernetes Pod restart count, lastState, node resource headroom.

Whether memory pressure shifted to other services on same node or downstream dependencies.

Stop rollout and rollback if any: node PSI or paging significantly worsens, other services latency rises, OOM persists, memory curve still unbounded, insufficient available replicas during rollout, or new limit causes scheduling failures.

13. Postmortem & Long-Term Governance

A complete postmortem must record: terminator, limit boundary, killed object, fault timeline, direct trigger, business root cause, fix verification, rollback result. Long-term governance recommendations:

Monitor node, parent cgroup, service, and container memory simultaneously; not just node utilization.

Increase sampling precision for key memory metrics; retain memory.events, PSI, working set, RSS, anon pages, restart reasons. Prometheus metrics per actual exporter.

Set explicit caps on caches, connection pools, task concurrency, per-request max memory.

Establish baselines with real load; set limits based on working set, peaks, safety margin — not arbitrary doubling.

Preserve kernel logs and container previous-instance logs; unify time across node, app, monitoring, release systems.

Prepare versioned manifests, single-instance canary, capacity checks, auto-rollback conditions for memory changes.

"Server still has memory" only describes host's aggregated state at a moment. Whether OOM occurs depends on which boundary the allocation hits, the pressure at that time, limits, reclaim capability, and policy. Separate global OOM, cgroup OOM, application OOM, and user-space policy kills; then use logs, counters, and high-resolution metrics to build an evidence chain — only then can you truly explain why the process was killed and how to prevent it next time.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

KubernetescgroupNUMAPSIsystemd-oomdLinux OOMcgroup v2memory troubleshooting
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.