Mastering awk: The Ultimate Guide to Linux Text Processing

Comprehensive Guide to the awk Command in Linux

1. Introduction to awk

awk is a powerful text‑processing language used for analyzing and manipulating text files. Its name comes from the initials of its creators: Alfred Aho, Peter Weinberger, and Brian Kernighan.

In Linux, awk is considered one of the "three musketeers" alongside grep and sed, each serving distinct text‑handling roles.

2. Purpose of awk

awk is primarily used for formatting and processing text data.

3. awk Syntax

awk [options] 'pattern {action}' file

4. Basic Usage

|     | awk is a language for processing text files.

5. Common Parameters

-F 'delimiter'   # Specify field separator (default is whitespace)

Example:

awk -F ':' '{print $NF}' /etc/passwd

6. awk Execution Lifecycle

awk reads input line by line, applies the specified rules, and after processing all lines, executes the END{} block.

7. Predefined Variables

$0

: the entire current line $n: the nth field NF: number of fields in the current line NR: record number (line number) FS: input field separator (default whitespace) OFS: output field separator (default space)

Examples:

awk -F ':' '{print $0, "---"}' /etc/passwd

awk -F ':' '{print $1}' /etc/passwd

awk -F ':' '{print NF}' /etc/passwd

awk -F ':' '{print NR}' /etc/passwd

awk 'BEGIN{FS=":"}{print $NF, $1}' /etc/passwd

8. Execution Flow of Rules

awk processes blocks in the order: BEGIN{} → pattern/action blocks → END{}. At least one block must be present, up to four.

9. Functions

print

: print output printf: formatted printing (e.g., %s for strings, %d for numbers, width specifiers)

Example:

awk -F ':' 'BEGIN{OFS=" | "}{printf "%+15s|%-15s
", $NF, $1}' /etc/passwd

10. Field Positioning

Use regular expressions to match lines, e.g., print lines containing "root".

awk -F ':' '/root/{print $0}' /etc/passwd

11. Comparison Expressions

>

: greater than <: less than >=: greater than or equal <=: less than or equal ~: matches regex !~: does not match regex

Example: print lines where group ID > user ID.

awk -F ':' '$4 > $3 {print $0}' /etc/passwd

12. Conditional Expressions

Use NR == 3 to print the third line.

awk -F ':' 'NR == 3 {print $0}' /etc/passwd

13. Logical Expressions

&&

: logical AND ||: logical OR !: logical NOT

Example combining conditions:

awk -F ':' '$3 + $4 > 2000 && $3 * $4 > 2000 {print $0}' /etc/passwd

14. Arithmetic Expressions

+

addition - subtraction * multiplication / division % modulus

Example: print lines where the sum of fields 3 and 4 exceeds 2000.

awk -F ':' '$3 + $4 > 2000 {print $0}' /etc/passwd

15. Flow Control

Awk supports if, for, and while constructs.

awk -F ':' '{if($3 > $4) print "greater"; else print "less or equal"}' /etc/passwd

awk -F ':' '{for(i=10;i>0;i--) print $0}' /etc/passwd

awk -F ':' '{i=1; while(i<10){print $0, i; i++}}' /etc/passwd

16. Practical Example

Print a separator line every five lines.

awk -F ':' '{if(NR%5==0){print "----------"} print $0}' /etc/passwd