Master Linux Text Processing: grep, sed, and awk Essentials

The Linux “three swordsmen” – grep, sed, and awk – are three essential command‑line tools for powerful text searching, stream editing, and data extraction, widely used for data analysis in shell scripts.

grep

grep is a powerful text‑search utility that finds lines matching a given pattern in files and outputs them. It supports many options and regular‑expression patterns for flexible searching.

Command format:

grep "search_string" filename

Basic options

Common options include: -c: count matches -i: ignore case -n: show line numbers

Example

Count occurrences of “xx” in text.log:

grep -c "xx" text.log

sed

sed is a stream editor for non‑interactive text transformations on input streams or files. It can search, insert, delete, and replace text, making bulk text processing efficient.

Basic concept

sed processes input line by line in a pattern space, applying commands such as a (append), i (insert), c (change), d (delete), y (transliterate), r (read file), and s (substitute).

Options

Common options: -n (quiet mode), -e (multiple commands), -i (in‑place editing), -f (script file), -r (extended regex).

Commands

Key commands: a – append after a line i – insert before a line c – replace entire line(s) d – delete lines y – transliterate characters = – print line numbers r – read file content s – substitute using regex.

Examples

Append “A” after each line in message: sed 'a A' message Insert “A” before line 1‑2: sed '1,2i A' message Replace all lines with “A”: sed 'c A' message Delete all lines: sed 'd' message Transliterate a→A, b→B: sed 'y/aB/Ab/' message Print line numbers for lines 1‑2:

sed '1,2=' message

awk

awk is a text‑processing language for pattern scanning and processing, ideal for extracting fields, performing calculations, and generating reports.

Basic concept

awk reads input into records and fields, using regular expressions and actions. It supports variables, arrays, and C‑like syntax.

Command format

awk [options] 'pattern {action}' file

Common options

-F: set input field separator

-v: assign a variable

-f: read program from a file

Built‑in variables

FS – input field separator

OFS – output field separator

NF – number of fields in the current record

NR – total number of input records

FILENAME – name of the current file

Examples

Print the second column of process list: ps | awk '{print $2}' Print first and second columns of /etc/passwd separated by a space: awk -F ":" '{print $1,$2}' /etc/passwd Print lines starting with “d” and their first column:

docker ps | awk '/^d/ {print $1}'