Master Linux Text Processing: grep, awk, and sed Essentials

Linux 三剑客

grep

-n: print line number
-A: after – print N lines after a match
-B: before – print N lines before a match
-C: center – print N lines before and after a match
-E: extended regex (e.g., grep -E 'root|nginx' /etc/passwd)
-v: invert match
-o: print only the matching part
-w: whole‑word match
-P: Perl‑compatible regex
-i: ignore case
-r: recursive search (e.g., grep -r 'www.baidu.com' ./)
-R: recursive including symlinks
-l: list matching file names only
-h: suppress file name prefix in output
-f: read patterns from file (useful for comparisons)
-c: count matching lines (like wc -l)
-G: basic regex
-m: stop after N matches (like head -n)

Example usage:

# grep -r 'www.zls.com' /root/web/
# grep -lr 'www.zls.com' /root/web/ | xargs sed -i.zls 's#www.zls.com#www.baidu.com#g'

awk

sed

In sed the core operations are divided into four categories:

增 – add

删 – delete

改 – modify

查 – search

Advanced topics include pattern space and hold space.

sed command execution flow

Example file zls.txt:

# zls.txt content
1,zls,666
2,wls,777
3,cls,888
4,lls,999

Running sed -n '3p' zls.txt prints the third line. The execution steps are:

sed reads the file line by line.

For each line it checks whether the line satisfies the condition.

If the -n option is set, it skips default output.

If the condition matches, the specified action (e.g., p, d, s) is performed.

After processing, the line is output only if not suppressed.

Processing continues until the end of the file.

sed – 查 (search)

sed command option

option meaning

sed command action

action meaning

-n

cancel default output

p

print

-r

enable extended regex

d

delete

a

append

i

insert

# sed -n '3p' zls.txt
1,zls,666
2,wls,777
3,cls,888
3,cls,888
4,lls,999

sed – 删 (delete)

# Delete line 2 (no file modification)
sed '2d' zls.txt

# Delete the last line
sed '$d' zls.txt

# Delete from line containing "zls" to line containing "cls"
sed -n '/zls/,/cls/d' zls.txt

sed – 增 (add)

# Replace entire line 2
sed '2c2,huanglong,438' zls.txt

# Append a line after the last line
sed '$a5,huanglong,438' zls.txt

# Insert a line before the first line
sed '1i2,huanglong,438' zls.txt

sed – 改 (modify)

# Substitute "zls" with "ZLS" globally
sed 's#zls#ZLS#g' zls.txt

# Replace any digit with 666 globally
sed 's#[0-9]#666#g' zls.txt

# Use back‑references
ifconfig eth0 | sed -nr 's#^.*inet (.*)  net.*#\1#gp'
ip a | sed -nr 's#^.*inet (.*)/24.*#\1#gp'

sed pattern space

Replace all newline characters in a file with spaces using the N command.

N   # read the next line into the pattern space

awk

awk built‑in variables, options, and actions

awk built‑in variable

meaning

awk option

option meaning

awk action

action meaning

NR

Number of Record (line number)

-F

field separator

gsub

global substitution

RS

Record separator (default \n)

-v

assign variable

print

output

FS

Field separator (default space)

NF

Number of fields in the current record

Note: In awk, output variables use single quotes, while bash variables use double quotes.

awk execution flow

The processing consists of three stages:

Before reading files – BEGIN{} block runs after options are parsed.

While reading files – pattern/action blocks are applied line by line.

After reading files – END{} block runs.

# Example: print first field of each line
awk '{print $1}' file.txt

# Example with BEGIN and END
awk 'BEGIN{print "name uid"}{print $1,$3}END{print "done"}' /etc/passwd | column -t

awk line and column operations

Lines are called records (NR), columns are fields (FS). Common examples:

# Print the first line
awk 'NR==1' /etc/passwd

# Print lines 1 to 3
awk 'NR>=1 && NR<=3' /etc/passwd

# Print lines matching a pattern
awk '/zls|cls/' zls.txt

awk column extraction

# Use ':' as field separator and print first column
awk -F: '{print $1}' /etc/passwd

# Print first and last column
awk -F: '{print $1,$NF}' /etc/passwd

# Change output separator to '#'
awk -F: -vOFS='#' '{print $1,$2,$3}' /etc/passwd

awk arrays

Arrays are useful for counting and grouping.

# Count occurrences of each IP in an nginx log
zcat access.log.gz | awk '{ip[$1]++; bytes[$1]+=$11} END {for(i in ip) print i, ip[i], bytes[i]}'

# Count occurrences of each domain in a URL list
awk -F/ '{domain[$3]++} END {for(d in domain) print d, domain[d]}' urls.txt

awk conditional statements

# Disk usage alert (threshold 70%)
df -h | awk -F '[ %]+' 'NR==2 {if($5>70) print "Disk space low"; else print "Disk space OK, usage:" $5 "%"}'

The tutorial provides a comprehensive overview of grep, awk, and sed, covering common options, practical command examples, and advanced usage patterns for text searching, extraction, transformation, and reporting on Linux systems.