Master Awk: Unlock Powerful Text Processing on the Command Line

Introduction

Awk, along with sed and grep, forms the trio of classic Unix text‑processing utilities. While all can match text, sed is a non‑interactive stream editor and awk is a pattern‑matching programming language that supports variables, functions, loops, and more.

What Awk Can Do

Treat a text file as a database of fields and records.

Use variables in text‑database operations.

Perform arithmetic and string manipulation.

Employ programming constructs such as conditionals and loops.

Format output and define custom functions.

Execute Unix commands within an Awk script.

Process the output of Unix commands.

Command‑Line Syntax

Awk supports two forms of command‑line syntax:

awk [-F ERE] [-v assignment] ... program [argument ...]
awk [-F ERE] -f progfile ... [-v assignment] ... [argument ...]

The program consists of pattern‑action pairs, similar to a script. Options -F, -v, and -f define the field separator, variable assignments, and external script files respectively.

Records and Fields

By default a record is a line separated by \n, and fields are separated by whitespace. These defaults can be changed with the built‑in variables RS (record separator) and FS (field separator). Access fields with $1, $2, … and the whole record with $0. The variable NF holds the number of fields.

Script Structure

A script is a series of pattern { action } blocks. If the pattern is omitted, the action runs for every input line. Example: echo -e "line1\nline2" | awk '{print}' Functions can be defined with the function name(parameters) { statements } syntax, and called like any other command.

Patterns

Patterns include regular expressions, relational expressions, the special BEGIN and END patterns, and pattern pairs. Logical operators && and || can combine patterns.

Regular Expressions

Awk uses extended regular expressions (ERE). For detailed regex syntax, refer to POSIX documentation.

Expressions and Operators

Awk supports arithmetic, string, and logical operators. See the man page for the full precedence table.

Statements

Common statements are print, printf, delete, break, continue, exit, and next. The next statement skips to the next record, similar to sed 's n command.

Built‑in Variables

Awk provides many built‑in variables such as ARGC, ARGV, FS, RS, NF, NR, FNR, FILENAME, ENVIRON, CONVFMT, OFMT, RSTART, and RLENGTH. These control input parsing, output formatting, and environment interaction.

Functions

Mathematical Functions

Awk includes atan2, cos, sin, exp, log, sqrt, int, rand, and srand. Example of generating a random integer between 0 and n‑1:

function randint(n) { return int(n*rand()); }
BEGIN { srand(); print randint(10); }

String Functions

Functions such as sub, gsub, index, length, match, split, sprintf, substr, tolower, and toupper provide powerful text manipulation capabilities.

I/O Functions

getline

reads input from files, commands, or standard input; close closes opened files or pipes; system executes external commands.

Conclusion

This guide, based on the Awk manual and the "Sed & Awk" appendix, offers a solid overview of Awk's features and should help readers begin writing effective Awk scripts for text processing tasks.