Why Does OOM Occur Even When Server Memory Looks Sufficient?

1. Identify the OOM Type

Global OOM : the whole host cannot reclaim memory, kernel OOM killer terminates a process.

cgroup OOM : a service or container exceeds its memory limit even though the host still has free memory.

Kubernetes container OOMKilled : the container exceeds its limit or its cgroup OOM, kubelet records the reason as OOMKilled.

systemd‑oomd : a user‑space daemon that kills cgroups based on PSI pressure, not the kernel OOM killer.

Application‑level OOM : e.g., Java OutOfMemoryError, database internal limits, malloc failures – may not involve kernel killing.

External signal termination : deployment scripts, probes, or admins may send SIGKILL (exit code 137) which is not always OOM.

2. Collect Evidence – Preserve the Facts

Record the failure timestamp, host, process or pod, application version, recent changes, and monitoring interval. Verify the current node and system time:

hostnamectl
date --iso-8601=seconds
uptime

Save a snapshot of memory status:

free -h
cat /proc/meminfo
vmstat 1 10
ps -eo pid,ppid,user,stat,rss,vsz,pmem,etime,comm,args --sort=-rss | head -n 30

For a specific PID (replace PID=12345 with the real PID):

cat "/proc/$PID/status"
cat "/proc/$PID/smaps_rollup"
cat "/proc/$PID/limits"

Check that the diagnostic directory has enough space and inodes before writing logs; do not delete OOM logs because they are essential evidence.

df -h /tmp
df -i /tmp

3. Understand Memory Metrics

The free command shows total, used, free, and cached memory, but the available column (MemAvailable) is a better indicator of memory that can be allocated without swapping. Situations where free -h shows plenty of memory yet OOM occurs include:

The OOM killer already terminated a large process, freeing memory after the event.

Monitoring sampled every minute missed a short‑lived memory spike.

cgroup OOM where MemAvailable reflects the whole node, not the cgroup's quota.

Container limits (MemoryMax=) or higher‑level cgroup limits restrict usage.

Allocation required a specific memory zone or contiguous pages that the node could not satisfy.

Therefore, you cannot invalidate OOM logs solely with a post‑mortem free -h output.

4. Check Kernel Logs – Global or memcg OOM

On systemd systems, query the relevant time window:

journalctl -k --since '2026-07-10 14:00:00' --until '2026-07-10 14:30:00' --no-pager

Or use a generic filter:

journalctl -k --since '-2 hours' --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

If the system lacks persistent journal, inspect /var/log/messages, /var/log/syslog, or the container’s dmesg (may be hidden by namespaces).

Kernel OOM records typically contain the task, gfp_mask, order, constraint info, memory list, and the final "Killed process" line. Look for:

Presence of oom-kill or Out of memory strings.

Constraint fields such as constraint, mems_allowed, nodemask indicating NUMA or memcg limits.

cgroup path or "Memory cgroup out of memory" messages.

Killed PID, total‑vm, anon‑rss, file‑rss, shmem‑rss, etc.

Memory reclaim, swap activity, and kernel warnings before and after the event.

5. Inspect cgroup – The Most Common "Host Has Memory, Yet OOM" Scenario

Determine the cgroup of the target process: cat "/proc/$PID/cgroup" Typical cgroup‑v2 files to read (replace CGROUP_PATH with the real path):

cat "$CGROUP_PATH/memory.current"
cat "$CGROUP_PATH/memory.max"
cat "$CGROUP_PATH/memory.high"
cat "$CGROUP_PATH/memory.events"
cat "$CGROUP_PATH/memory.stat"
cat "$CGROUP_PATH/memory.pressure"

Key fields: memory.current: current usage of the cgroup and its descendants. memory.max: hard limit ("max" means unlimited). memory.high: high‑watermark that triggers reclamation but is not a hard cap. memory.events: counters such as low, high, max, oom, oom_kill (kernel version dependent). memory.stat: breakdown of anonymous, file, kernel, slab pages. memory.pressure: PSI memory‑pressure metrics.

Note that memory.events shows cumulative counts; a single "oom_kill 3" line only proves three OOM events over the cgroup’s lifetime, not that one just occurred. Compare before/after snapshots.

Systemd service limits can be inspected with:

systemctl show example.service -p MemoryCurrent -p MemoryHigh -p MemoryMax -p MemorySwapMax -p OOMPolicy
systemctl cat example.service

Also check drop‑in configurations under /etc/systemd/system/example.service.d/*.conf because they may override the main unit.

Use systemd-cgls and systemd-cgtop to view hierarchical cgroup usage.

6. Verify OOMKilled in Kubernetes

Set the namespace, pod, and container variables, then run:

NAMESPACE=production
POD=example-pod
CONTAINER=example-container
kubectl -n "$NAMESPACE" get pod "$POD" -o wide
kubectl -n "$NAMESPACE" describe pod "$POD"
kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"
"}{end}'
kubectl -n "$NAMESPACE" logs "$POD" -c "$CONTAINER" --previous --timestamps

If the pod never restarted, --previous may error. Also inspect the pod spec for resource requests and limits:

kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{range .spec.containers[*]}{.name}{"\trequests="}{.resources.requests.memory}{"\tlimits="}{.resources.limits.memory}{"
"}{end}'
kubectl -n "$NAMESPACE" get events --field-selector involvedObject.name="$POD" --sort-by='.metadata.creationTimestamp'

Events have a retention period; lack of events does not mean no OOM. Full evidence should include the container’s lastState.terminated.reason, node kernel/runtime logs, cgroup memory.events deltas, and the container’s memory curve.

Prometheus metrics useful for diagnosis include container_memory_working_set_bytes, container_memory_rss, container_memory_cache, and container_memory_failcnt. Their semantics depend on the exporter and cgroup version.

If a pod has no memory limit, it can still be killed by a node‑wide OOM; the pod’s QoS class and oom_score_adj affect the selection but do not guarantee protection.

7. systemd‑oomd – User‑Space OOM Handling

Check the service status and logs:

systemctl status systemd-oomd --no-pager
journalctl -u systemd-oomd --since '-2 hours' --no-pager
oomctl   # if available (depends on systemd version)

systemd‑oomd bases its decisions on cgroup‑v2 and PSI. Relevant unit properties include ManagedOOMSwap= and ManagedOOMMemoryPressure=. Consult man systemd.resource-control for the exact supported options.

If logs show systemd‑oomd killed a cgroup, the root cause is not a kernel‑level memory exhaustion; you must still verify which higher‑level cgroup enabled the policy, how long the pressure lasted, swap usage, and whether the service’s memory growth matches the workload.

Disabling systemd‑oomd to avoid accidental kills is discouraged because it removes a safety net and can turn controllable OOMs into full‑system hangs.

8. Distinguish Application OOM from Kernel OOM

Java OutOfMemoryError can stem from heap, metaspace, direct buffers, or thread creation and may not be caused by the kernel killer. Even if the Java heap is below the container limit, native memory, thread stacks, JIT, and shared libraries count toward the container’s usage. When the process is still alive, use JDK tools:

jcmd "$PID" VM.flags
jcmd "$PID" GC.heap_info
jcmd "$PID" VM.native_memory summary

Native Memory Tracking must be enabled at JVM start; otherwise the last command provides limited data. Heap dumps can cause pauses and heavy I/O, so avoid them on low‑disk‑space or production nodes.

Other runtimes (Go, Node.js, Python, databases) have their own memory models; use the appropriate diagnostic interfaces and align the findings with cgroup and kernel data.

9. Global OOM – Where Did the Memory Go?

Inspect /proc/meminfo fields such as AnonPages, Shmem, SReclaimable, SUnreclaim, Slab, PageTables, HugePages_Free, Committed_AS, and CommitLimit (availability depends on kernel config). For slab growth, use: slabtop -o Tmpfs consumes memory; list and check usage:

findmnt -t tmpfs
df -h -t tmpfs

Open deleted tmpfs files with lsof +L1 (limit scope on large hosts).

Check swap and PSI:

swapon --show
vmstat 1 10
cat /proc/pressure/memory

In vmstat, the si and so columns show swap‑in/out rates; PSI some and full indicate memory‑pressure stalls. Compare against historical baselines.

Inspect overcommit settings:

sysctl vm.overcommit_memory vm.overcommit_ratio
grep -E 'CommitLimit|Committed_AS' /proc/meminfo

NUMA awareness is essential; check allowed nodes and current binding:

numactl --hardware
numastat
numastat -p "$PID"
cat "/proc/$PID/status" | grep -E 'Mems_allowed|Cpus_allowed'

10. Derive Root Cause from Evidence, Not Guesswork

For cgroup OOM, you need matching memory.current / memory.max values, an increase in memory.events, and supporting kernel logs. Memory leaks require a sustained growth of anonymous memory or native heap under the same load. Spikes need high‑resolution memory curves. Global OOM needs kernel logs, MemAvailable, swap, PSI, and a degradation of reclaim. systemd‑oomd actions need its own logs and policy evidence.

Distinguish the immediate trigger (e.g., memory.max breach) from the business root cause (unbounded cache, increased batch concurrency, mismatched limits).

11. Fixes – Control Risk Before Changing Limits

1. Temporary Control

Use existing application throttling, concurrency limits, task pausing, cache caps, or traffic shedding. Verify queue length, idempotency, and SLA impact before pausing batch jobs.

2. Adjust Container or systemd Memory Limits

Increasing limits triggers a rolling update in Kubernetes Deployments; for systemd services, changing MemoryMax= may require daemon-reload and a service restart.

Before changing limits, assess impact scope, current working set, node free memory, QoS class, PDB, replica count, and rollout strategy.

Impact scope : target pod/service, parent slice, other workloads on the node.

Pre‑change checks : current working set, peak usage, node allocatable memory, remaining capacity, QoS, PDB, replica count.

Backup : save original manifests, Helm values, systemd unit files, OOM logs, and metric snapshots.

Canary : apply changes to a test environment or a single replica/node, observe at least one full business cycle.

Verification : ensure memory.events stops growing, working set and RSS stay bounded, PSI normalizes, error rate and latency do not regress, and node memory remains sufficient.

Rollback : revert to the previous manifest or release version if node pressure worsens or pods fail to schedule.

3. Increase or Adjust Swap

Creating a swap file involves mkswap, swapon, and updating /etc/fstab. Verify disk space, filesystem support, encryption requirements, and cluster policies before enabling swap. Validate with swapon --show, free -h, vmstat, PSI, and application latency. Never disable swap ( swapoff) when memory is already scarce.

4. Tune Kernel Parameters

Parameters such as vm.overcommit_memory, vm.overcommit_ratio, vm.swappiness, HugePages, and NUMA settings can affect allocation behavior. Record current values, test under realistic load, and apply changes via sysctl -w (temporary) or persistent sysctl files. Rollback must restore both runtime and persisted values.

12. Verification and Rollback Criteria

Check memory.current vs memory.max and that memory.events no longer increases.

Node MemAvailable, swap activity, PSI, slab, and page‑table metrics should be stable.

Application working set, anonymous memory, heap, and native memory should stay within expected bounds.

Request volume, concurrency, error rate, throughput, and P95/P99 latency should recover.

Kubernetes pod restart count, lastState, and node resource headroom should be healthy.

Ensure no new memory pressure is transferred to other services on the same node.

If any of the above regress, stop scaling and roll back immediately.

13. Post‑mortem and Long‑Term Governance

A complete post‑mortem should record the terminator, limit boundary, killed object, timeline, direct trigger, business root cause, fix verification, and rollback outcome.

Monitor node, parent cgroup, service, and container memory simultaneously—not just node usage.

Increase sampling granularity for memory.events, PSI, working set, RSS, anonymous pages, and OOM reasons.

Set explicit limits for caches, connection pools, task concurrency, and per‑request memory.

Build baselines under realistic load and set limits based on observed working set and safety margin.

Preserve kernel logs and previous container logs; synchronize timestamps across nodes, applications, monitoring, and release systems.

Version control memory‑change manifests, perform single‑instance canary, capacity checks, and define automatic rollback conditions.

Only by correlating logs, counters, and high‑resolution metrics can you explain why a process was killed and prevent future OOM incidents.