Why Did Opening a Log with Vim Kill the Java Process?

Incident Timeline

At 15:19 a port‑exception alarm fired for service data‑stream‑openapi. When the host was inspected, the Java process had vanished. The immediate response was to cut traffic to the container, rebuild it, and redeploy; the service recovered by 15:26.

Initial Investigation

After the service was back online, the root cause was pursued. Monitoring via Odin showed a sudden memory‑usage spike at 15:19 followed by a rapid drop, but no Full GC events were recorded. YGC activity rose briefly at 15:25, likely due to the container restart, and CPU usage remained stable except for a short rise during the rebuild.

Finding the OOM Kill

Running dmesg -T | grep java revealed that the Java process (PID 1155805) was killed at 15:17 by the Linux OOM killer with the reason “out of memory”. The port alarm had been configured to trigger after 12 consecutive 10‑second failures, linking the OOM event to the observed alarm.

Vim’s Role

A colleague reported that they had opened the nginx log with vim. The log file was 37 GB, far exceeding the container’s 8 GB memory limit. Vim reads the entire file via its readfile function (which internally calls the read system call), causing the container to exhaust memory and invoke the OOM killer.

Understanding Vim and the OOM Killer

Vim loads a file completely into memory; opening a huge file can be extremely slow and memory‑intensive.

The Linux OOM killer activates when memory is critically low and selects a victim based on a scoring algorithm. In this case the Java process received a higher score than the vim process, so it was terminated.

/* OOM killer comment from kernel source */
* If we run out of memory, we have the choice between either
* killing a random task (bad), letting the system crash (worse)
* OR try to be smart about which process to kill. Note that we
* don’t have to be perfect here, we just have to be good.

Operational Recommendations

Do not use vim to view large log files in production containers. Prefer lightweight tools such as less, grep, or tail, and always verify file size before opening it with an editor.