Master Linux Text Processing: Pipes, grep, sed & awk Explained

This chapter introduces Linux’s pipe, grep, sed, and awk commands, explaining their syntax, options, and practical usage examples.

1. Pipe Command

The pipe (|) passes the output of one command as input to another, enabling powerful command chaining.

Basic Syntax

command1 | command2

Multiple pipes can be chained, e.g., command1 | command2 | command3.

Examples

Show processes sorted by memory usage: ps aux --sort=-%mem | less Count lines containing specific text: grep 'specific text' filename | wc -l List only directories: ls -l | grep '^d' Count error messages (case‑insensitive):

grep -i 'error' logfile.log | wc -l

Notes

Each command runs in its own subshell; variables are not shared.

Input and output are handled via stdin and stdout unless redirected.

Standard error is not passed through the pipe unless explicitly redirected (e.g., 2>&1).

2. grep Command

grep searches files for patterns using regular expressions and prints matching lines.

Basic Syntax

grep [options]... pattern [file]...

Common Options

-i

: ignore case -v: invert match -c: count matching lines -l: list matching file names -n: show line numbers -r or -R: recursive search -E: use extended regular expressions -F: treat pattern as a fixed string -o: output only the matching part -A NUM: show NUM lines after a match -B NUM: show NUM lines before a match -C NUM: show NUM lines of context

Examples

Basic search: grep pattern file.txt Ignore case: grep -i pattern file.txt Show line numbers: grep -n pattern file.txt Recursive search: grep -rnw '/path/to/directory' -e 'pattern' Colorized output:

grep 'match_pattern' file_name --color=auto

3. sed Command

sed is a stream editor that processes text line by line, applying editing commands without modifying the original file unless redirected.

Basic Syntax

sed [options]... 'command' [input-file]...

Common Options

-e

: add an editing command -f: read commands from a file -i: edit files in place -n: suppress automatic printing -r: enable extended regular expressions -u: update atomically (used with -i)

Typical Commands

p

: print matching lines d: delete matching lines s/old/new/g: substitute globally i\text: insert text before a line a\text: append text after a line c\text: replace the whole line

Examples

Print first five lines: sed -n '1,5p' filename Delete empty lines: sed '/^$/d' filename Replace “old” with “new”: sed 's/old/new/g' filename In‑place substitution: sed -i 's/old/new/g' filename Insert a line before line 3: sed '3i\New line' filename Append a line after line 3: sed '3a\New line' filename Replace lines 2‑5 with a single line:

sed '2,5c\This is a new line' filename

4. awk Command

awk is a line‑oriented programming language for pattern scanning and processing.

Basic Syntax

awk [options] 'pattern {action}' [filename]

Common Options

-F fs

: set field separator -v var=val: assign variable -f script: read program from file -i inplace: edit files in place (GNU awk) -n: suppress default output

Built‑in Variables

ARGC , ARGV : command‑line arguments

FILENAME , FNR , NR : file and record counters

FS , OFS : input and output field separators

RS , ORS : record separators

NF : number of fields in the current record

$0 , $1 …: whole line and individual fields

Typical Usage

Print all lines: awk '{print}' file.txt Print first column: awk '{print $1}' file.txt Sum values in the first column: awk '{sum+=$1} END {print sum}' numbers.txt Filter lines matching a pattern: awk '/pattern/ {print}' file.txt Use a custom field separator:

awk -F: '{print $1}' /etc/passwd

Notes

awk processes input line by line, default field separator is any whitespace.

It supports variables, control structures (if, for, while), and a rich set of string and math functions.

BEGIN and END blocks run before the first line and after the last line, respectively.

For more details, see the original article at https://www.cnblogs.com/xyh9039/p/18337347 . © Original author.