Why a Container Engine Restart Can Kill Your App: FIFO Bug Explained

Problem Overview

A large customer reported that an application inside a container stopped responding to HTTP requests. Initial checks of container network connectivity showed no issues, leaving the team puzzled.

Investigation

After reviewing recent changes, the team discovered that the container engine had been upgraded and restarted the night before. Observing the pod logs, they noticed that the container’s standard output also stopped, and the process received a SIGPIPE signal.

Root Cause

The restart broke the FIFO used by the container engine to forward stdio. The FIFO’s write end was opened with O_WRONLY, so when the buffer filled it generated a broken‑pipe error and the process could no longer respond.

FIFO Basics

Opening a FIFO in write‑only mode without a read end causes EPIPE (Broken pipe) when the buffer overflows. Using read‑write mode ( O_RDWR) prevents this issue.

Fix

Changing the flag from O_WRONLY to O_RDWR in the code that opens the FIFO resolves the problem. A short code snippet can be used to verify the fix:

pouch run -d nigix

Container IO Flow

The article then explains the full stdio forwarding chain in a container runtime stack (pouch + containerd + runc/kata):

Process stdio → pipe end → shim process pipe → shim FIFO write end → pouch FIFO write end → log destination (e.g., JSON file).

Images illustrate how pouch creates the container, how containerd’s cio package opens FIFOs, and how the shim process copies stdout/stderr to the FIFO.

Conclusion

The incident demonstrates that what appears to be a network issue can actually stem from low‑level stdio handling. Understanding the FIFO mechanics and the container runtime’s IO pipeline enables quick identification and resolution of similar problems.