Why Did My Linux Server Kill My Java Process at 5 AM? Understanding the OOM Killer

At 6 am the author was woken up by an early alarm caused by a failure in the Plumbr service, which was detected by their monitoring system.

After checking the application logs and finding no errors, the author examined the kernel log and discovered OOM killer messages indicating that the Java process (PID 29957) was killed due to out‑of‑memory conditions.

Jun407:41:59 plumbr kernel:[70667120.897649] Out of memory: Kill process 29957 (java) score 366 or sacrifice child
Jun407:41:59 plumbr kernel:[70667120.897701] Killed process 29957 (java) total-vm:2532680kB, anon-rss:1416508kB, filers:0kB

The Linux OOM killer activates when the system runs out of memory, selecting a victim process based on a heuristic score. By default the kernel allows memory overcommit, and on the author's EC2 m1.small instance swap was disabled, which together with a traffic spike caused the Java service to exhaust memory.

Understanding the OOM Killer

Linux permits processes to request more memory than is physically available; most programs never use all allocated memory, similar to how ISPs oversell bandwidth. When memory becomes scarce, the OOM killer is triggered and kills the process with the lowest score.

What Triggered It?

The kernel parameter /proc/sys/vm/overcommit_memory was set to 1, allowing every malloc() request to succeed.

The service ran on an EC2 m1.small instance with swap disabled.

These factors, combined with a sudden traffic surge, caused the Java process to continuously request memory until the OOM killer terminated it.

Reproducing the Issue

The following Java program repeatedly allocates large int arrays until the OOM killer kills the process, reproducing the same kernel messages.

package eu.plumbr.demo;
public class OOM {
    public static void main(String[] args){
        java.util.List l = new java.util.ArrayList();
        for (int i = 10000; i < 100000; i++) {
            try {
                l.add(new int[100_000_000]);
            } catch (Throwable t) {
                t.printStackTrace();
            }
        }
    }
}

Running this on Ubuntu with a 2 GB heap (‑Xmx2g) and no swap produces the same “Out of memory: Kill process (java) …” log.

Swap Configuration Used for Testing

swapoff -a
dd if=/dev/zero of=swapfile bs=1024 count=655360
mkswap swapfile
swapon swapfile

Solutions

In the author's case the service was moved to a machine with more RAM. Enabling swap was considered but rejected due to poor JVM GC performance on swap. Other options include tuning the OOM killer, distributing load across multiple smaller instances, or reducing the application’s memory footprint.