Comprehensive Guide to Using AWK for Text Processing

AWK is a powerful text‑processing language that excels at handling and extracting data from structured or semi‑structured files.

Basic Usage

First create a sample file log.txt :

2 this is a test
3 Do you like awk
This's a test
10 There are orange,apple,mongo

Usage 1 – Matching rule and action

The core syntax is awk 'pattern{action}' file . Example:

# Print the first and fourth fields of each line
$ awk '{print $1,$4}' log.txt
---------------------------------------------
2 a
3 like
This's
10 orange,apple,mongo
# Formatted output using printf
$ awk '{printf "%-8s %-10s\n",$1,$4}' log.txt
---------------------------------------------
2        a
3        like
This's
10       orange,apple,mongo
# Print a blank line for each empty line
$ awk '/^$/ {print "Blank line"}' log.txt
---------------------------------------------
Blank line

Usage 2 – Specifying field separator

Use -F or the built‑in variable FS to change the delimiter.

# Use comma as separator
$ awk -F, '{print $1,$2}' log.txt
---------------------------------------------
2 this is a test
3 Do you like awk
This's a test
10 There are orange apple
# Use multiple delimiters
$ awk -F'[ ,]' '{print $1,$2,$5}' log.txt
---------------------------------------------
2 this test
3 Are awk
This's a
10 There apple

Usage 3 – Setting variables

# Set variable a to 1
$ awk -va=1 '{print $1,$1+a}' log.txt
---------------------------------------------
2 3
3 4
This's 1
10 11

Usage 4 – Using an external script file

$ awk -f cal.awk log.txt

Operators

Operator

Description

= += -= *= /= %= ^= **=

Assignment

?:

C conditional expression

||

Logical OR

&&

Logical AND

~ and !~

Match and non‑match regular expressions

< <= > >= != ==

Relational operators

+

Addition

-

Subtraction

* / %

Multiplication, division, remainder

++ --

Increment / decrement

$

Field reference

in

Array membership

Built‑in Variables

Variable

Description

$n

n‑th field of the current record

$0

Entire input record

ARGC

Number of command‑line arguments

ARGV

Array of command‑line arguments

FS

Input field separator (default whitespace)

OFS

Output field separator

NR

Number of records read so far

FNR

Record number in the current file

RS

Record separator (default newline)

ORS

Output record separator

IGNORECASE

If true, matching is case‑insensitive

Additional Examples

Print lines where the second column contains “th”:

$ awk '$2 ~ /th/ {print $2,$4}' log.txt
this a

Case‑insensitive search:

$ awk 'BEGIN{IGNORECASE=1} /this/' log.txt
2 this is a test
This's a test

Calculate total size of all *.txt files:

$ ls -l *.txt | awk '{sum+=$5} END {print sum}'
666581

Print a multiplication table:

seq 9 | sed 'H;g' | awk -v RS='' '{for(i=1;i<=NF;i++)printf("%dx%d=%d%s", i, NR, i*NR, i==NR?"\n":"\t")}'

These snippets demonstrate how AWK can be combined with other Unix utilities to perform complex data‑processing tasks efficiently.