Mastering sed and awk: Essential Linux Text‑Processing Tools Explained

Introduction

This article focuses on two essential Linux tools— sed and awk —which are indispensable for any Linux power user.

What is sed

According to the book sed and awk , sed is a non‑interactive stream editor that processes input character streams and outputs the transformed result, making it extremely useful for command‑line text manipulation.

Basic Concepts of sed

The basic syntax is: sed [-nefr] [action] Key options: -n: quiet mode; only explicitly printed lines are output. -e: execute the following editing command. -f: read editing commands from a file. -r: enable extended regular expressions. -i: edit files in place.

Common actions (used inside [n1],[n2]action syntax): a: append text after the addressed line. c: replace the addressed lines with new text. d: delete the addressed lines. i: insert text before the addressed line. p: print the addressed line (usually with -n). s: substitute pattern with replacement using regular expressions.

Typical sed Usage

Example 1 – Delete lines 2 to 5

nl /etc/passwd | sed '2,5d'

Deletes lines 2 through 5.

Quotes are required around the action.

Example 2 – Append text after line 2

nl /etc/passwd | sed '2a drink tea'

Adds "drink tea" after line 2 (i.e., as line 3).

Use 2i to insert before line 2.

Example 3 – Append multiple lines

nl /etc/passwd | sed '2a drink tea or ......\
drink beer'

Uses a backslash to continue the appended text on a new line.

Replacing and Displaying Lines

Example 1 – Replace lines 2‑5

nl /etc/passwd | sed '2,5c No 2-5 number'

Example 2 – Print lines 5‑7 only

nl /etc/passwd | sed -n '5,7p'

Example 3 – Simple substitution

sed 's/old/new/g'

Combining sed with grep and regular expressions to extract an IP address

/sbin/ifconfig eth0 | grep 'inet addr' | sed 's/^.*inet addr://g' | sed 's/Bcast.*$//g'

This pipeline isolates the IP address from the output of ifconfig.

In‑place File Editing (use with caution)

Example – Replace trailing periods with exclamation marks

sed -i '/s\.$/!/g' test.txt

Example – Append a comment line at the end of a file

sed -i '$a # This is a test' test.txt

What is awk

awk is a data‑processing language that operates on a per‑line basis, splitting each line into fields for flexible manipulation.

Basic Concepts of awk

Typical syntax:

awk 'condition1{action1} condition2{action2} ...' filename

Actions are placed inside single quotes and curly braces.

awk can read from files or from standard input of another command.

Default field separator is whitespace (space or tab).

Example – Print the first and third fields of the last five login entries:

last -n 5 | awk '{print $1 "\t" $3}'

awk Operators

Greater than: > Less than: < Greater or equal: >= Less or equal: <= Equal: == Not equal:

!=

Example – List users with UID less than 10

cat /etc/passwd | awk '{FS=":"} $3 < 10 {print $1 "\t" $3}'

Example – Same as above using BEGIN to set the field separator

cat /etc/passwd | awk 'BEGIN {FS=":"} $3 < 10 {print $1 "\t" $3}'

Example – Summing columns and formatting output

cat test.txt | awk 'NR==1{printf "%10s %10s %10s %10s %10s
",$1,$2,$3,$4,"Total"}
NR>1{total=$2+$3+$4; printf "%10s %10d %10d %10.2f
", $1,$2,$3,$4,total}'

Key points:

Multiple commands inside an action can be separated by semicolons or newlines.

Use == for equality tests. printf requires \n for line breaks.

Variables in awk are referenced directly without a leading $.