Why Does Your Linux Pipe Output Lag? Real‑Time Flow and SIGPIPE Explained

Definition of a Pipe

A pipe is a kernel‑managed circular buffer that connects the standard output of one process to the standard input of another. The buffer has a fixed size (typically 64 KB on modern Linux). When the buffer is empty, a read() blocks; when the buffer is full, a write() blocks. The pipe disappears automatically when both ends are closed.

Real‑time data flow in a pipeline

To observe whether data is transferred incrementally, create a simple Python script that prints two strings with a three‑second pause:

# 1.py
import time
while True:
    print('1111')
    time.sleep(3)
    print('2222')
    time.sleep(3)

Run it with a consumer such as cat: python3 1.py | cat Because Python’s standard output is fully buffered when it is redirected to a pipe, no output appears until the internal buffer (default 4 KB) is flushed. This demonstrates that the pipe itself does not delay data; the delay comes from the writer’s buffering strategy.

Buffering modes

Full buffering – data is written only when the buffer fills (common for regular files).

Line buffering – the buffer is flushed on each newline character (default for terminals).

No buffering – data is written immediately (standard error).

Making the script output immediately

Two practical ways to force the data to reach the pipe as soon as it is produced:

Method 1 – Fill the buffer

import time
while True:
    print('1111' * 4096)   # 4096 bytes ≈ pipe buffer size
    time.sleep(3)
    print('2222' * 4096)
    time.sleep(3)

This writes a block large enough to fill the pipe’s buffer, causing the kernel to unblock the writer and deliver the data to the consumer.

Method 2 – Explicit flush

import sys, time
while True:
    print('1111')
    sys.stdout.flush()
    time.sleep(3)
    print('2222')
    sys.stdout.flush()
    time.sleep(3)

Calling sys.stdout.flush() forces the buffered data to be written to the pipe after each line, so the right‑hand command receives it immediately.

SIGPIPE and broken‑pipe handling

When a process writes to a pipe whose read end has been closed, the kernel sends the writer a SIGPIPE signal. The default action is to terminate the process, and the write system call fails with EPIPE. This behaviour can be observed with the following script:

# writer.py
import time, sys
while True:
    time.sleep(10)
    print('1111')
    sys.stdout.flush()

Execute it in one terminal: python3 writer.py | cat Then, in another terminal, locate the cat process and kill it (e.g., kill <pid_of_cat>). The writer terminates with a traceback similar to:

Traceback (most recent call last):
  File "writer.py", line 6, in <module>
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Terminated

Key observations:

The writer receives SIGPIPE only when it attempts a write after the reader has disappeared.

If the writer never writes again, it continues running silently.

Pipe read/write rules (POSIX summary)

If no data is available, a blocking read() waits until data arrives; with O_NONBLOCK it returns –1 and sets errno to EAGAIN.

If the pipe is full, a blocking write() waits until space is freed; with O_NONBLOCK it returns –1 and sets errno to EAGAIN.

When all write ends are closed, read() returns 0 (EOF).

When all read ends are closed, a subsequent write() generates SIGPIPE and fails with EPIPE.

Writes of size ≤ PIPE_BUF (usually 4096 bytes) are atomic; larger writes may be interleaved with data from other writers.

Conclusion

Linux pipes transmit data as soon as the writer flushes its output; the perceived delay is caused by the writer’s buffering mode. Understanding full, line, and no buffering, as well as the behaviour of SIGPIPE, allows developers to design pipelines that react predictably in scripts and command‑line workflows.