Master the Linux “Three Musketeers”: grep, sed, and awk Explained

Linux "three musketeers" are the three most important commands—grep, sed, and awk—known for their powerful functionality and wide application in shell data analysis.

1. grep

grep is a powerful text search tool that finds specified strings in file contents and outputs matching lines. It supports many options and pattern matching for flexible and efficient searches.

Command format:

grep "search_string" filename

Basic options

-c: count occurrences

-i: ignore case

-n: show line numbers

Example

Count occurrences of "xx" in text.log:

grep -c "xx" text.log

2. sed

sed is a stream editor used for basic text transformations on input streams or files. It can search, add, delete, and replace text non-interactively, improving efficiency for large text processing.

Basic concept

sed processes input line by line, applying commands. Syntax:

sed [options] 'command' filename

Common options

-n: quiet mode (suppress automatic printing)

-e: execute multiple commands

-i: edit files in place

-f: read commands from a script file

-r: enable extended regular expressions

Key commands (a, i, c, d, y, =, r, s)

a – append text after a line.

# Append "A" after every line
sed 'a A' message
# Append "A" after lines 1-2
sed '1,2a A' message
# Append multiple lines
sed '1,2a A
B
C' message

i – insert text before a line.

# Insert "A" before every line
sed 'i A' message
# Insert before lines 1-2
sed '1,2i A' message

c – replace entire lines with new text.

# Replace all lines with "A"
sed 'c A' message
# Replace lines 1-2 with "A"
sed '1,2c A' message

d – delete lines.

# Delete all lines
sed 'd' message
# Delete lines 1-2
sed '1,2d' message

y – transliterate characters.

# Replace a with A and b with B
sed 'y/aB/Ab/' message

= – print line numbers.

# Print line numbers for lines 1-2
sed '1,2=' message

r – read a file and insert its contents after a line.

# Insert contents of r.txt after line 1
sed '1r r.txt' message

s – substitute text using regular expressions.

# Replace all 'b' with 'x'
sed 's/b/x/g' message

3. awk

awk is a text processing tool for pattern scanning and data manipulation. It works on structured data, offering C‑like syntax for printing, variable handling, and arithmetic.

Basic concept

Typical syntax:

awk [options] 'pattern {action}' file

Common options

-F: set input field separator

-v: define or modify an awk variable

-f: read awk program from a file

Built‑in variables

FS – input field separator (default space/tab)

OFS – output field separator (default space)

NF – number of fields in the current record

NR – total record number

FILENAME – name of the current file

Examples

Print the second column of the output of ps: ps | awk '{print $2}' Print first and second columns of /etc/passwd separated by a space: awk -F ':' '{print $1,$2}' /etc/passwd Print lines starting with “a”:

awk '/^a/ {print $1}' file