Why Does OOM Occur Even When Server Memory Looks Sufficient?

1. Identify Which Kind of OOM Occurred

Global OOM : The whole host struggles to reclaim memory and the kernel OOM killer terminates a process.

cgroup OOM : A service or container exceeds its own memory limit even though the host still has free memory.

Kubernetes OOMKilled : The container exceeds its memory limit or the cgroup OOM; kubelet records the reason as OOMKilled.

systemd‑oomd : A userspace daemon that selects a cgroup based on pressure information and sends a kill signal; it is not the kernel OOM killer.

Application‑level memory errors : e.g., Java java.lang.OutOfMemoryError, database internal limits, or malloc failures, which may not involve kernel‑initiated kills.

External termination : Signals sent by deployment systems, probes, administrators, or process managers; exit code 137 usually means SIGKILL but not necessarily OOM.

The first goal is not to tweak parameters immediately but to confirm the terminator and the limit boundary.

2. Collect On‑Site Information – Preserve Evidence

Record the failure timestamp, host, process or Pod, application version, recent changes, and monitoring sampling interval. Verify the current node and system time:

hostnamectl
date --iso-8601=seconds
uptime

Save a memory snapshot:

free -h
cat /proc/meminfo
vmstat 1 10
ps -eo pid,ppid,user,stat,rss,vsz,pmem,etime,comm,args --sort=-rss | head -n 30

Note that ps RSS and VSZ are only for a quick filter; shared pages cause double counting. For detailed mapping, read:

PID=12345
cat "/proc/$PID/status"
cat "/proc/$PID/smaps_rollup"
cat "/proc/$PID/limits"

If the process has already exited, /proc/$PID disappears; then consult kernel logs, service logs, monitoring, and cgroup counters. When writing output to a file, first check the diagnostic directory capacity and inode usage. Do not delete OOM logs because they are essential evidence.

3. Correct Interpretation – free Is Not available

Running free -h shows low free memory, which is normal because Linux uses idle memory for page cache and can reclaim it when needed. The more useful field is MemAvailable, an estimate of memory that can be allocated without swapping, but it is still only an estimate.

Scenarios where “available is high but OOM happened” include:

The OOM killer terminated a large process, freeing memory afterward.

Monitoring samples every minute missed a short‑lived memory spike.

A cgroup OOM occurred; the host’s MemAvailable does not reflect the cgroup’s limit.

The node shows total memory, but the container’s MemoryMax= or other limits applied.

The allocation required a specific memory zone or contiguous pages that the global available metric cannot guarantee.

Therefore, you cannot dismiss OOM logs solely based on post‑mortem free -h output.

4. Check Kernel Logs – Confirm Global or memcg OOM

On systemd systems, query the relevant time window:

journalctl -k --since '2026-07-10 14:00:00' --until '2026-07-10 14:30:00' --no-pager

Or filter for common keywords:

journalctl -k --since '-2 hours' --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

If the system does not persist journal logs, inspect /var/log/messages, /var/log/syslog, or the logging platform. Inside containers, dmesg may be unavailable due to namespace restrictions.

Kernel OOM records usually contain the offending task, gfp_mask, order, constraint info, memory list, and the final Killed process. Fields vary by kernel version, so do not rely on a single regex. Important fields to look for:

Presence of oom-kill or Out of memory strings.

Constraints such as constraint, mems_allowed, nodemask indicating cpuset, NUMA, or memcg limits.

Memory cgroup path in the log.

Killed PID, process name, and memory statistics like total-vm, anon-rss, file-rss, shmem-rss.

Pre‑ and post‑failure memory reclaim, swap activity, kernel warnings, and process start records.

Note that memory.events counters are cumulative; a line like oom_kill 3 only shows three occurrences over the cgroup’s lifetime, not that a new OOM just happened. Compare increments before and after the incident.

5. Inspect cgroup – The Most Common “Host Has Memory, Yet Application OOMs”

Determine which cgroup the target process belongs to: cat "/proc/$PID/cgroup" For cgroup v2, the unified hierarchy path can be used to read relevant files (replace placeholders with real values):

CGROUP_PATH=/sys/fs/cgroup/system.slice/example.service
cat "$CGROUP_PATH/memory.current"
cat "$CGROUP_PATH/memory.max"
cat "$CGROUP_PATH/memory.high"
cat "$CGROUP_PATH/memory.events"
cat "$CGROUP_PATH/memory.stat"
cat "$CGROUP_PATH/memory.pressure"

Key fields: memory.current: Current memory usage of the cgroup and its descendants. memory.max: Hard limit; value max means no hard limit. memory.high: High‑water mark that triggers reclamation but is not a hard cap. memory.events: Counters such as low, high, max, oom, oom_kill (kernel version dependent). memory.stat: Breakdown of anonymous pages, file pages, kernel memory, slab, etc. memory.pressure: PSI memory‑pressure information.

When memory.events shows a rise in oom_kill, compare the increment with the failure window. Also verify parent slice limits ( MemoryMax=) because they can affect multiple services.

6. Kubernetes – Verify OOMKilled

All placeholders ( $NAMESPACE, $POD, $CONTAINER) must be replaced with real values. Typical commands:

NAMESPACE=production
POD=example-pod
CONTAINER=example-container
kubectl -n "$NAMESPACE" get pod "$POD" -o wide
kubectl -n "$NAMESPACE" describe pod "$POD"
kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"
"}{end}'
kubectl -n "$NAMESPACE" logs "$POD" -c "$CONTAINER" --previous --timestamps

If the container never restarted, --previous may error out. Also check the resource limits in the workload manifest:

kubectl -n "$NAMESPACE" get pod "$POD" -o jsonpath='{range .spec.containers[*]}{.name}{"\trequests="}{.resources.requests.memory}{"\tlimits="}{.resources.limits.memory}{"
"}{end}'
kubectl -n "$NAMESPACE" get events --field-selector involvedObject.name="$POD" --sort-by='.metadata.creationTimestamp'

Events have a retention period; lack of events does not mean no OOM. Full evidence should include the container’s lastState.terminated.reason, node kernel or runtime logs, cgroup memory.events delta, and the container’s memory curve.

Common container metrics (Prometheus) include container_memory_working_set_bytes, container_memory_rss, container_memory_cache, and container_memory_failcnt. Exporter versions and collector configurations affect semantics; always use the actual exposed metric.

If a Pod has no memory limit, it can still be killed by a global node OOM. Kubernetes QoS class and oom_score_adj influence the selection under node pressure but do not guarantee protection.

7. systemd‑oomd – Not Kernel OOM, Yet Can Kill Services

Check the service status and logs:

systemctl status systemd-oomd --no-pager
journalctl -u systemd-oomd --since '-2 hours' --no-pager
oomctl   # availability depends on systemd version

systemd‑oomd uses cgroup v2 and PSI to decide kills. Its unit can be configured with ManagedOOMSwap=, ManagedOOMMemoryPressure=, etc.; consult man systemd.resource-control for the exact version.

If logs show systemd‑oomd killed a cgroup, the root cause is not a kernel memory exhaustion. Identify which parent cgroup enabled the policy, how long the pressure persisted, swap usage, and whether the service’s memory growth matches the workload.

Do not disable systemd‑oomd merely to avoid accidental kills; disabling changes the host‑wide memory‑pressure protection and may turn a controllable OOM into a full system stall.

8. Separate Application OOM from Kernel OOM

Java OutOfMemoryError can stem from heap, metaspace, direct buffers, or thread creation and does not necessarily involve the kernel OOM killer. Even if the Java heap is below the container limit, native memory, thread stacks, JIT, and shared libraries count toward the container’s memory.

When the process is still alive and JDK tools are available, inspect:

jcmd "$PID" VM.flags
jcmd "$PID" GC.heap_info
jcmd "$PID" VM.native_memory summary

Native Memory Tracking must be enabled at JVM start; otherwise the last command provides incomplete data. Dumping the heap can cause pauses and heavy I/O; avoid it on low‑disk‑space or production peaks. Other runtimes (Go, Node.js, Python) have their own memory models and diagnostics; align their data with cgroup and kernel metrics.

If the application reports allocation failures but the kernel shows no OOM, also examine the process’s own limits ( /proc/$PID/limits).

9. Global OOM – Where Did the Memory Go?

Inspect /proc/meminfo for fields such as AnonPages, Shmem, SReclaimable, SUnreclaim, Slab, PageTables, HugePages_Free, Committed_AS, and CommitLimit (presence depends on kernel config). To view slab usage: slabtop -o If SUnreclaim or a particular slab grows continuously, correlate with time series, kernel version, and workload; a single large value does not prove a leak. Tmpfs also consumes memory:

findmnt -t tmpfs
df -h -t tmpfs

Deleted but still‑open tmpfs files are not freed immediately; locate them with lsof +L1 (limit the search on large hosts).

Check swap and PSI:

swapon --show
vmstat 1 10
cat /proc/pressure/memory

vmstat

columns si and so show pages swapped in/out per second; existing swap usage does not equal frequent swapping. PSI some and full indicate task stalls due to memory pressure and should be compared against historical baselines. Absence of swap does not guarantee safety, and presence of swap does not solve a persistent leak. Kubernetes node swap support varies by version.

Check overcommit settings:

sysctl vm.overcommit_memory vm.overcommit_ratio
grep -E 'CommitLimit|Committed_AS' /proc/meminfo

Committed_AS

is not the current RSS. Changing overcommit affects databases, caches, and large forks; it is a high‑risk kernel change. For NUMA hosts, also verify allowed memory nodes:

numactl --hardware
numastat
numastat -p "$PID"
cat "/proc/$PID/status" | grep -E 'Mems_allowed|Cpus_allowed'

Do not modify NUMA or overcommit parameters without evaluating latency and bandwidth impact.

10. Derive Root Cause From Evidence, Not From Result

Root cause must be supported by evidence from the same time window. For cgroup OOM, you need memory.current / memory.max, memory.events delta, and kernel logs. For memory leaks, you need a consistent growth of anonymous memory or heap under the same load, plus allocation‑path evidence. For short spikes, you need high‑resolution concurrency and memory curves. For global OOM, you need kernel logs, MemAvailable, swap, PSI, and reclamation degradation. For systemd‑oomd, you need its logs and policy evidence.

Distinguish trigger conditions from business root causes. A limit breach ( memory.max) is the immediate trigger; the business cause may be unbounded cache, increased batch concurrency, or mismatched limit versus working set.

11. Fix Plan – Control Risk First, Then Adjust Limits

1. Temporary Control

Prefer using existing application throttling, concurrency limits, task pausing, cache caps, or traffic shedding. Pausing batch jobs or lowering concurrency may affect data freshness and backlog; verify queue length, idempotency, recovery speed, and SLA before proceeding.

2. Adjust Container or systemd Memory Limits

Raising limits is not free. Changing a Kubernetes Deployment’s resources triggers a rolling update; modifying MemoryMax= in a systemd unit may require daemon-reload and service restart.

Impact Scope : Target Pod, service, parent slice, and remaining node memory.

Pre‑check : Current working set, peak, node allocatable memory, remaining capacity, QoS, PDB, replica count, rollout strategy.

Backup : Save original manifests, Helm values, systemd unit and drop‑ins, OOM logs, and metric snapshots.

Canary : Test in a staging environment or single replica/node, limit traffic, observe at least one full business cycle.

Verification : Ensure memory.events stops growing, working set and RSS stay bounded, PSI normal, error rate and latency unchanged, node headroom sufficient.

Rollback : Restore previous manifest or prior release and roll back immediately if node pressure spikes.

3. Add or Adjust Swap

Create a swap file, run mkswap and swapon, and update /etc/fstab. This changes host memory behavior and is high‑risk. Verify disk space, filesystem support, encryption, Kubernetes/kubelet policies, and application tolerance to swap latency. Backup /etc/fstab and current swapon --show output, test on a similar node, then roll out gradually.

Validation should include swapon --show, free -h, vmstat, PSI, application P95/P99 latency, and node reboot state. Do not swapoff when memory is scarce, as it may trigger another OOM.

4. Modify Kernel Parameters

Parameters such as vm.overcommit_memory, vm.overcommit_ratio, vm.swappiness, HugePages, and NUMA settings can affect overall workload. Record original values, confirm vendor recommendations, perform load testing on identical nodes, and roll out via configuration management with a gradual canary. Temporary sysctl -w changes differ from persistent files; rollback must restore both runtime and persisted values.

12. Verification and Rollback Criteria

Check memory.current vs. memory.max distance and that memory.events no longer increments.

Node MemAvailable, swap activity, PSI, slab, and page‑table stability.

Application working set, anonymous memory, heap, and native memory reach a stable plateau.

Request volume, concurrency, error rate, throughput, and P95/P99 latency recover.

Kubernetes Pod restart count, lastState, and node resource headroom.

No new memory pressure transferred to other services on the same node.

If any of the above regress, stop scaling and roll back immediately.

13. Post‑mortem and Long‑Term Governance

Record terminator, limit boundary, killed object, timeline, direct trigger, business root cause, fix verification, and rollback result. Long‑term recommendations:

Monitor node, parent cgroup, service, and container memory simultaneously—not just node usage.

Increase sampling granularity for key metrics; retain memory.events, PSI, working set, RSS, anonymous pages, and restart reasons. Use the actual exporter’s metrics.

Set explicit caps for caches, connection pools, task concurrency, and per‑request max memory.

Build baselines under real load; define limits based on working set, peaks, and safety margin rather than arbitrary multiples.

Persist kernel logs and previous container logs; synchronize timestamps across nodes, applications, monitoring, and release systems.

Version control memory‑related manifests, perform single‑instance canaries, capacity checks, and automated rollback conditions.

“The server shows free memory” only describes a snapshot. Whether OOM occurs depends on where the allocation happens, the pressure at that moment, limits, reclamation ability, and policies. Distinguish global OOM, cgroup OOM, application OOM, and user‑space policy termination, then build an evidence chain from logs, counters, and high‑resolution metrics to truly explain why a process was killed and how to prevent it next time.