How Linux Pipes Achieve Real‑Time Data Flow: Buffering, SIGPIPE & Tips

Linux users are familiar with the pipe operator "|" that connects commands, but many assume the left‑hand command finishes completely before its output reaches the right‑hand command.

Pipes operate simultaneously: the left command writes to the pipe and the right command reads and processes the data immediately.

Definition of a pipe

A pipe is a kernel‑managed buffer, essentially a circular memory region. One end is attached to a process's stdout (writer) and the other to a process's stdin (reader). When the buffer is empty, a read blocks until data arrives; when full, a write blocks until space is freed. The pipe disappears automatically when both ends close.

In an experiment with COMMAND1 | COMMAND2, the output of COMMAND1 is bound to the pipe’s write end and COMMAND2 reads from the pipe’s read end, so data is transferred as soon as COMMAND1 produces it.

Observations showed that the program appeared to hang because of I/O buffering. Linux provides three buffering modes:

Full buffering – data is written only when the buffer fills (common for files).

Line buffering – data is flushed on newline (standard output).

No buffering – data is written immediately (standard error).

Python’s default stdout is buffered; when redirected to a pipe it uses full buffering unless flush() is called or the buffer fills. Two code variants were shown (images) to demonstrate flushing the output.

Results confirmed that as soon as COMMAND1 writes to the pipe (whether the buffer is full or flush() is called), COMMAND2 receives and processes the data immediately.

SIGPIPE and broken pipe

The signal SIGPIPE (or the condition “broken pipe”) occurs when a process writes to a pipe whose read end has been closed. The article lists the read/write rules:

When no data is available, read() blocks unless O_NONBLOCK is set, in which case it returns -1 with errno=EAGAIN.

When the pipe is full, write() blocks unless O_NONBLOCK is set, then it returns -1 with errno=EAGAIN.

If all write‑end file descriptors are closed, read() returns 0 (EOF).

If all read‑end file descriptors are closed, a write() generates SIGPIPE.

Writes of size ≤ PIPE_BUF are atomic; larger writes are not guaranteed to be atomic.

An experiment demonstrated SIGPIPE: the reader process was killed before the writer wrote, causing the writer to receive SIGPIPE and terminate only when it attempted the actual write.

Summary

The article explains that Linux pipes transmit data in real time, governed by kernel buffering and signal handling, and provides practical tips for using pipes effectively in shell scripting and programming.