Master Linux Redirection and Pipes: From File Descriptors to Inter‑Process Communication

Redirection

In computer technology, redirection means changing the direction or target of a data stream; in Linux it is commonly used to control command input and output.

Linux defines three predefined data streams: standard output (stdout), standard input (stdin) and standard error (stderr). Users interact with processes through these streams.

Standard input (stdin): when a program needs input (e.g., user‑provided data), it reads from stdin, typically the keyboard in a terminal.

Standard output (stdout): the main output of a program is sent to the user via this stream.

Standard error (stderr): a separate stream for error messages or warnings, allowing users to distinguish regular output from problems.

In Linux, stdin corresponds to file descriptor 0, stdout to 1, and stderr to 2.

File descriptor : a non‑negative integer assigned by the OS to each opened file or resource, acting as a pointer that lets the system track and manipulate the file.

A process has a default limit on the number of open file descriptors; when the limit is reached, opening new files fails. The limit can be increased temporarily with

ulimit -n number

or permanently by editing

/etc/security/limits.conf

:

<code># Temporary effect
ulimit -n number

# Persistent effect
vim /etc/security/limits.conf
# <username>  [hard|soft] nofile  <new_limit>
# * means all users
# soft: warning only
# hard: cannot open more files
# <new_limit>: new limit, "unlimited" for no limit</code>

In Linux, file descriptor 0 is linked to

/dev/stdin

, 1 to

/dev/stdout

, and 2 to

/dev/stderr

. Redirection symbols can change where these descriptors read from or write to.

Standard input redirection uses the

1>

symbol.

Standard output redirection uses the

<

symbol.

Standard error redirection uses the

2>

symbol.

Redirection Applications

Example: redirect screen output to a file

<code>echo "12345" &gt; 1.txt
# Redirects the output that would appear on the screen into 1.txt</code>

Example: redirect stdout and stderr to different files

<code>output1 &gt; file1  2 &gt; file2</code>

Example: redirect both stdout and stderr to the same file

<code>output 1&gt; file_name 2&gt;&amp;1
# Short form
outpu &amp;&gt; file_name</code>

Notes:

Using

&>

or

>&

combines stdout and stderr.

2>&1

redirects stderr into the same destination as stdout.

The order matters when combining streams.

Pipes

In Linux, a pipe is a communication mechanism that allows two or more processes to transfer data, passing the output of one process as the input of another, similar to water flowing through a pipe.

Pipe characteristics:

FIFO (first‑in‑first‑out) ordering of data.

Half‑duplex: only one end can write or read at a time.

Read operations block when the pipe is empty; write operations block when the pipe is full.

Memory‑based, usable only between processes on the same machine.

The write end belongs to one process, the read end to another, enabling data flow from one process to the next.

Inter‑Process Communication

A process is an instance of a running program with its own isolated address space. Because processes are isolated, they must use mechanisms such as shared memory or pipes to communicate.

Anonymous Pipe

The

|

symbol creates an anonymous pipe via the shell, allowing communication between parent‑child processes.

Anonymous pipe workflow:

The shell creates an anonymous pipe using system calls.

The commands on both sides of the pipe are executed as child processes of the shell.

The left‑hand command writes its output to the pipe’s write end.

The right‑hand command reads from the pipe’s read end.

Example:

<code>cat /etc/passwd | wc -l</code>

Named Pipe

Create a named pipe with:

<code>mkfifo pipe_name</code>

Named pipes persist in the filesystem and can be used for communication between any two processes, unlike anonymous pipes which exist only for the duration of the parent‑child relationship.

Use cases:

Anonymous pipe: useful for temporary data transfer between processes in a single command or script, without needing a persistent connection.

Named pipe: suitable when a long‑lived, persistent communication channel between multiple processes is required.

Pipe Behavior Illustration

Data stored in a pipe blocks the reading process until it is consumed, after which the pipe becomes unblocked.

If a pipe is empty, a read operation blocks until data becomes available.