Master Linux Pipelines and Redirection: Practical Commands and Real-World Examples

Introduction

In a Linux command‑line environment, pipelines (|) and redirection are powerful tools that can dramatically improve the efficiency and flexibility of command execution. This guide explores advanced usage of these features with detailed code examples.

Basic Usage of Pipelines

Simple Command Combination

Connect two or more commands so that the output of one becomes the input of the next.

# Example: view a log file and filter for "error"
cat /var/log/syslog | grep "error"

Multi‑Stage Pipelines

Chain multiple commands to create more complex data‑processing flows.

# Example: list files and sort by size
ls -l | sort -k5

Basic Redirection

Redirect Standard Output to a File

Save a command’s standard output for later review.

# Example: save directory listing to list.txt
ls > list.txt

Redirect Error Output

Capture error messages to a file for debugging.

# Example: save error messages to error.log
command_that_might_fail 2> error.log

Combining Pipelines and Redirection

Filter with grep and Save Results

Use a pipeline to filter data and then redirect the filtered output.

# Example: filter log entries and save to result.txt
cat /var/log/syslog | grep "pattern" > result.txt

Redirect Output of Multiple Commands

Store the combined result of several commands in a file.

# Example: list files, filter with grep, and save
ls -l | grep "keyword" > output.txt

Using tee for Simultaneous Output

The tee command writes to both the screen and a file.

# Example: display and save directory listing
ls -l | tee output.txt

Real‑World Scenarios

Live Log Monitoring

Combine tail with a pipeline to filter log entries in real time.

# Example: monitor syslog for "error" entries
tail -f /var/log/syslog | grep "error"

Batch File Processing

Use find to locate files and pipe the list to other commands.

# Example: copy all .txt files to a backup directory
find . -name "*.txt" | xargs -I {} cp {} /backup

Using awk for Text Manipulation

awk

can extract and reformat columns, especially when combined with pipelines.

# Example: extract columns 2 and 3 from data.txt
cat data.txt | awk '{print $2,$3}' > extracted_data.txt

Using xargs for Bulk Operations

xargs

reads items from standard input and passes them as arguments to another command.

# Example: delete all .txt files found by find
find . -name "*.txt" | xargs rm

Combining Pipelines, Redirection, awk , and xargs

Count Keyword Occurrences in Logs

Chain multiple tools to count how often a keyword appears and store the result.

# Example: count "error" occurrences and save
cat /var/log/syslog | grep "error" | awk '{print $4}' | sort | uniq -c > error_count.txt

Custom Script Integration

Processing Data with a User Script

Feed pipeline data into a custom script for more complex logic.

# Example: process log file with a custom script
cat /var/log/syslog | my_custom_script.sh > processed_data.txt

Process Substitution

Use process substitution to treat command output as a temporary file.

# Example: compare two generated files without creating them
diff <(cat file1.txt) <(cat file2.txt)

Conclusion

By mastering pipelines, redirection, and related utilities such as tee, awk, xargs, custom scripts, and process substitution, you can handle complex data‑processing tasks efficiently in the Linux shell, greatly enhancing productivity in everyday operations.