Master Linux Redirection, File Descriptors, and Pipes: A Practical Guide

Redirect

In computer technology, redirection means changing the direction or target of a data flow; in Linux it is commonly used to control command input and output.

Linux defines three predefined data streams: standard input (stdin), standard output (stdout) and standard error (stderr). Users interact with processes through these streams.

Standard input (stdin): when a program needs input (e.g., user data), it reads from stdin, typically the keyboard in a terminal.

Standard output (stdout): the main output of a program is sent to the user via this stream.

Standard error (stderr): a separate stream for error messages or warnings, allowing users to distinguish normal output from problems.

In Linux, stdin corresponds to file descriptor 0, stdout to 1, and stderr to 2.

File descriptors: A non‑negative integer assigned by the OS to each opened file or resource, acting as a pointer that lets the system track and manipulate the file.

Each process has a limit on the number of open file descriptors; when the limit is reached, opening new files fails. The limit can be increased temporarily with ulimit -n number or permanently by editing /etc/security/limits.conf.

# Temporary effect
ulimit -n number

# Persistent effect
vim /etc/security/limits.conf
# <username>  [hard|soft] nofile  <new_limit>
# * means all users, soft = warning, hard = enforce, unlimited = no limit

In Linux, file descriptor 0 is linked to /dev/stdin, 1 to /dev/stdout, and 2 to /dev/stderr. Redirection symbols can change where these descriptors point.

Redirect standard input with 1> (note: the original text had a mistake; actually stdin uses < ).

Redirect standard output with > .

Redirect standard error with 2> .

Redirection examples:

echo "12345" > 1.txt  # send output to a file

output 1> file1 2> file2  # send stdout to file1 and stderr to file2

output 1> file_name 2>&1  # combine stdout and stderr into one file
outpu &> file_name          # shorthand

Notes:

The & symbol indicates that the following number is a descriptor, not a filename.

When combining stdout and stderr, the order matters: the correct descriptor must be specified first.

Incorrect example: output 2>&1 > file.txt This fails because stderr is redirected to stdout before stdout knows its own target, causing the output to go to the screen instead of the file.

Pipe

A pipe in Linux is a communication mechanism that allows data to flow from one process to another, similar to water flowing through a pipe.

Pipe characteristics:

Data is transferred in a FIFO (first‑in‑first‑out) order.

It is half‑duplex: only one end can write or read at a time.

Reading blocks when the pipe is empty; writing blocks when the pipe is full.

It is memory‑based and works only between processes on the same machine.

The write end belongs to one process, the read end to another, enabling one process’s output to become another’s input.

Inter‑process communication (IPC): Processes are independent execution entities with isolated address spaces. To exchange data they use shared memory or pipes. Pipes create a unidirectional data channel for communication.

Linux supports two pipe types: unnamed (anonymous) pipes and named pipes.

Unnamed pipe: used between parent and child processes that share a lineage.

Named pipe: can be used between any two processes.

Anonymous pipe: The | symbol creates an anonymous pipe via the shell. Workflow:

The shell creates the pipe using system calls.

The commands on either side of | run as child processes of the shell.

The left‑hand command writes its output to the pipe’s write end.

The right‑hand command reads from the pipe’s read end.

Example: cat /etc/passwd | wc -l Although cat and wc are not parent/child, the shell creates a pipe that connects them, allowing indirect IPC.

Named pipe:

Creation: mkfifo pipe_name The named pipe exists in the filesystem as a buffer for IPC and persists until removed.

Use cases:

Anonymous pipe: suitable for transient data transfer between processes in a single script or command line.

Named pipe: useful when a persistent connection between processes is required over a longer period.

Pipe behavior: Data written to a pipe blocks the writer when the pipe is full; reading blocks when the pipe is empty. Once data is read, the pipe becomes unblocked and can accept new data.