Master Text Processing with grep, sed, and awk: Practical Commands and Examples

grep

grep is a command‑line utility for searching plain‑text data sets for lines that match a regular expression.

Basic syntax

grep [OPTIONS]... PATTERN [FILE]...

Common options

-v

invert match (select non‑matching lines) -i ignore case -c print only a count of matching lines -n prefix each line of output with its line number ^PATTERN match lines that start with

PATTERN

PATTERN$

match lines that end with

PATTERN

^$

match empty lines

Examples

# grep moon monkey
"Oh! My god! The moon has fallen into the well!"
"Goodness me! The moon is really in the water!"
"The moon is in the well."
# grep -v moon monkey          # invert match
# grep -c moon monkey          # count matches (returns 8)
# grep -i my monkey            # ignore case
# grep -n monkey monkey       # show line numbers
# grep '^J' monkey            # lines starting with J
# grep '呢$' monkey            # lines ending with "呢"

sed

sed is a stream editor that can perform basic text transformations on an input stream (a file or input from a pipeline).

Basic syntax

sed [OPTIONS]... '{command}' [FILE]...

Frequently used commands

d

delete the selected line(s) s/REGEXP/REPLACEMENT/[FLAGS] substitute y/SET1/SET2/ transliterate characters i\TEXT insert TEXT before the current line a\TEXT append TEXT after the current line p print the current pattern space q quit processing

Substitution flags

n

replace only the n ‑th occurrence on the line g replace all occurrences \1, \2 … refer to captured groups & re‑use the entire matched text

Examples

# echo "Twinkle, twinkle, little star" | sed 's/little/big/'
Twinkle, twinkle, big star

# sed '2d' word               # delete the second line
# echo "hello" | sed 'i\kitty'
kitty
hello

# sed '2c\hello kitty' word   # replace the second line with "hello kitty"
# sed -n '/star/w c' word     # write lines containing "star" to file c
# sed '3q' word               # print first three lines then quit

awk

awk is a pattern‑scanning and processing language. It treats each input line as a record split into fields.

Invocation

awk [OPTIONS] -f program.awk [FILE]...
awk [OPTIONS] 'program' FILE...

Fields

Fields are accessed as $1, $2, …, $NF. $0 represents the whole line. The default field separator is whitespace; it can be changed with -F.

Built‑in variables

NF

number of fields in the current record NR total number of records read so far FS input field separator RS record separator FILENAME name of the current input file ARGC number of command‑line arguments

Common functions

length(s)

return length of string

s

index(s,t)

position of substring t in

s

match(s, r)

returns 1 if s matches regex

r

split(s, a, fs)

split s into array a using

fs

gsub(r, s)

replace all matches of regex r with

s

substr(s, p[, n])

substring of s starting at position p (optional length n)

Example data file students_store

Xiaoka   60 80 40 90 77 class-1
Yizhihua 70 66 50 80 90 class-1
kerwin   80 90 60 70 60 class-2
Fengzheng 90 78 62 40 62 class-2
xman     -  -  -  -  - class-3 php

Typical one‑liners

Print the whole file: awk '{print $0}' students_store Print only the name column: awk '{print $1}' students_store Rows where the 7th field equals “class‑1”: awk '$7=="class-1" {print}' students_store Rows where the 7th field is not “class‑1”: awk '$7!="class-1" {print}' students_store Numeric comparison (math score > 60): awk '$2>60 {print}' students_store Logical AND: awk '$2>60 && $3>60 {print}' students_store Logical OR: awk '$2>80 || $4>80 {print}' students_store Regex match: awk '$0 ~ /class-1/ {print}' students_store BEGIN/END blocks for headers and footers:

awk 'BEGIN {print "Name   Math   Chinese   English   History   Sport   grade"}
     {print}
     END {print "--- end of report ---"}' students_store

Formatted output with printf

awk '{printf "%‑10s %3d %3d %3d %3d %3d %s
", $1, $2, $3, $4, $5, $6, $7}' students_store

Awk script file example

#!/usr/bin/awk -f
BEGIN { print "my name is Java小咖秀" }

Make it executable ( chmod +x script.awk) and run it ( ./script.awk).