Why Did My Container Stop Responding? The Hidden FIFO Bug in Container Engines

Problem Overview

During a morning meeting a customer reported that an application inside a container could not respond to HTTP requests. Initial checks of container network and routing showed no issues, leading to further investigation.

Investigation Begins (Not a Network Issue)

After consulting the application team we learned the pod ran a standard Java service. A recent container‑engine upgrade had restarted the engine, which we initially dismissed.

Online troubleshooting suggested recreating the pod, but a colleague reproduced the issue offline, allowing deeper analysis.

Root Cause Identification

The application hung while trying to write logs to stdout; the container’s standard output stopped, and the process received a SIGPIPE. The container engine forwards stdio via a FIFO; after the engine restart the FIFO was broken, causing the write to fail with EPIPE.

Changing the FIFO open mode from O_WRONLY to O_RDWR resolved the issue.

FIFO Basics

Opening a FIFO only for writing (O_WRONLY) causes a write error (EPIPE) when the read end is not open or the buffer overflows. Using read‑write mode (O_RDWR) prevents the error.

Example to verify:

pouch run -d nginx

Container I/O Architecture

The article explains how container stdio is forwarded: the container’s process writes to a pipe, the shim reads the pipe and writes to a FIFO opened by containerd, which is then read by Pouch and logged.

Key components include Pouch, containerd, and runc (or Kata). Diagrams illustrate the flow of stdout/stderr through pipes and FIFOs.

Detailed Steps in Pouch, containerd, and runc

Pouch creates a task via containerd’s NewTask which initializes I/O by opening a FIFO. containerd’s cio package opens two FIFOs (stdout and stderr) in read‑write mode, then copies data from the shim’s pipe to the FIFO.

The shim creates OS pipes, passes one end to the container process and the other to the FIFO. runc creates the container’s init process, which inherits these file descriptors.

Conclusion

The incident demonstrates that what appears to be a network problem can stem from low‑level I/O handling. Systematic debugging, examining logs and code, and understanding the underlying mechanisms are essential.

Key takeaways: analyze symptoms, avoid assumptions about component stability, study the full stack, and respect all code.