Understanding Linux Pipe Real‑Time Data Flow and SIGPIPE Tips

Many Linux users are familiar with the pipe symbol "|" for chaining commands, but the real‑time nature of data flow through a pipe is often misunderstood.

Pipes operate simultaneously on both ends: as soon as the left‑hand command writes to the pipe, the right‑hand command can read and process the data.

Pipe Definition

A pipe is a kernel‑managed buffer, essentially a circular memory structure. One end connects to a process's output, the other to a process's input. When the buffer is empty, a read blocks until data arrives; when full, a write blocks until space is freed. The pipe disappears automatically when both ends close.

Thus, in a command like COMMAND1 | COMMAND2, COMMAND1's stdout is bound to the pipe's write end, and COMMAND2's stdin to the read end, so data is transferred immediately as it is produced.

Experiment Demonstrating Real‑Time Transfer

An experiment shows that the output from the first command is not delayed until it finishes; instead, the second command receives data as soon as it is written, unless buffering blocks the transfer.

File I/O Buffering Modes

Full buffering: data is written only when the buffer is full (common for files).

Line buffering: data is flushed on newline characters (standard output).

No buffering: data is written immediately (standard error).

Python’s default stdout is buffered; when redirected to a pipe, it typically uses full buffering, so output appears only after the buffer fills or after an explicit flush() call.

SIGPIPE and Broken Pipe

The signal SIGPIPE (or “broken pipe”) occurs when a process writes to a pipe whose read end has been closed.

Read/Write Rules:

When no data is available to read:

If O_NONBLOCK is not set, read blocks until data arrives.

If O_NONBLOCK is set, read returns -1 with errno=EAGAIN.

When the pipe is full:

If O_NONBLOCK is not set, write blocks until space is freed.

If O_NONBLOCK is set, write returns -1 with errno=EAGAIN.

If all write‑end file descriptors are closed, read returns 0 (EOF).

If all read‑end file descriptors are closed, a write triggers SIGPIPE .

Writes of size ≤ PIPE_BUF are atomic on Linux.

Writes larger than PIPE_BUF are not guaranteed to be atomic.

Demonstrations show that killing the read‑end process causes the writer to receive SIGPIPE and terminate, but the signal is delivered only when the writer actually attempts to write to the closed pipe.

Summary

Through theory and experiments, we see that Linux pipes transmit data instantly between commands, subject to buffering behavior, and that proper handling of read/write rules and SIGPIPE is essential for robust shell scripting.