Mastering awk: Powerful Text Processing Techniques for Linux

awk is a pattern‑scanning and processing tool that, compared with grep for searching and sed for editing, excels at data analysis and report generation. It reads one or more files line by line, matches patterns, splits lines using a field separator (default space), and then processes the fields.

Gawk

gawk (GNU awk) is the GNU version of awk. Linux users often link /bin/awk to /bin/gawk so that the awk command invokes gawk.

How to use awk

1. Command‑line usage: awk [-F field-separator] 'COMMAND' inputfiles where COMMAND is the awk program, -F specifies a custom field separator, and inputfiles are the files to process.

2. Shell‑script mode: write awk commands in a script file, start the file with #!/bin/awk, and execute the script by its name.

3. Using a separate program file: store awk commands in a file and run awk -f awk-script inputfiles.

Basic syntax awk [OPTION] 'program' FILE1 FILE2 … Program format: pattern { action }. Options include -F to set the input field separator and -v to assign variables.

Example of setting the separator to ':' and printing usernames and default shells:

awk -F: '{print $1,$7}' /etc/passwd

Assigning a variable with -v:

awk -v a="hello awk" '{print a}'

Variables can also be set in a BEGIN block, e.g.:

awk 'BEGIN{a="hello awk"}{print a}'

awk variables

Variables are divided into built‑in and user‑defined.

Built‑in variables (excerpt):

User‑defined variables can be set with -v on the command line or inside the program using BEGIN{var=value}.

Examples of common options: FS: input field separator (default space). OFS: output field separator. NR, FNR: record numbers. ARGC: number of arguments.

Pattern types include regular expressions, expressions, ranges, and special BEGIN / END patterns.

Example of a regular‑expression pattern:

awk -F: '/root/ {print $1,$7}' /etc/passwd

Example of an expression pattern:

awk -F: '$3>=500 {print $1,$2,$3}' /etc/passwd

Common operators and their symbols are shown in the following tables:

Redirection:

Output redirection uses > (overwrite) or >> (append).

Input redirection can be performed with the getline function, which reads from a command, a pipe, or another file.

Example:

awk 'BEGIN{"date" | getline d; print d}'

Printing functions print outputs items separated by spaces; printf formats output using a format string.

Examples:

awk 'BEGIN{print "line1","line2","line3"}'

awk -F: '{print $1,$3,$7}' /etc/passwd

Formatting with printf uses specifiers such as %s, %d, %f, etc., and modifiers for width, precision, alignment, and sign.

Example of left‑aligned fields:

awk -F: '{printf "%-15s %-20s
", $1, $7}' /etc/passwd

Control structures

Awk supports if‑else, while, do‑while, for, break, continue, and next statements.

Example of an if‑else test:

awk -F: '{if($3>=500) print $1 " is a common user"; else print $1 " is admin or system user"}' /etc/passwd

Looping over fields:

awk '{i=1; while(i<=NF){printf "%s ", $i; i+=2}; print ""}' /etc/inittab

Iterating over an array:

awk 'BEGIN{split("root:x:0:0", user, ":"); for(i in user) print user[i]}'

This overview summarizes the essential features of awk; further exploration can deepen understanding of its capabilities.