Mastering awk: Essential Syntax, Patterns, and Real-World Examples

Basic Syntax

The core awk syntax is awk 'pattern { action }' file, where pattern selects input lines, action defines what to do for each matching line, and file is the input source. pattern – a regular expression or condition used to match input data. action – the commands executed when the pattern matches. file – the text file to be processed.

Basic Examples

Print every line of a file

awk '{print}' file.txt

This prints each line of file.txt.

Print the first column of a file

awk '{print $1}' file.txt

This prints the first field of each line in file.txt.

Field Separator

By default awk splits fields on whitespace, but you can change the delimiter with the -F option.

Use a comma as the field separator

awk -F, '{print $1}' data.csv

This prints the first column of each line in data.csv, using a comma as the separator.

Conditional Matching

awk can filter data using pattern matching.

Print lines containing "error"

awk '/error/ {print}' log.txt

This outputs all lines from log.txt that contain the word "error".

Arithmetic Operations

awk supports arithmetic, allowing calculations on fields.

Sum numbers in a file

awk '{sum += $1} END {print sum}' numbers.txt

This computes the total of the first column in numbers.txt and prints the result after processing the file.

Built‑in Variables

awk provides several built‑in variables for handling input data.

Print the total number of lines

awk 'END {print NR}' file.txt

This prints the line count of file.txt.

Custom Functions

You can define reusable functions inside an awk script.

Calculate the average of a line

awk 'function average(arr) {
    total = 0;
    for (i = 1; i <= NF; i++) {
        total += $i;
    }
    return total / NF;
}
{print "Average: " average($0)}' data.txt

This defines an average function that computes the mean of all fields in a line and prints it.

Control Flow

awk supports conditional statements and loops for more complex processing.

Label numbers based on size

awk '{
    if ($1 < 10) {print "Small: " $1}
    else if ($1 < 50) {print "Medium: " $1}
    else {print "Large: " $1}
}' numbers.txt

This tags each number as Small, Medium, or Large depending on its value.

Processing Command Output

awk can also work with the output of other commands.

List filenames in the current directory

ls -l | awk '{print $9}'

This extracts and prints the ninth field (the filename) from the ls -l output.

Summary

awk is a versatile text‑processing utility for extracting, transforming, and reporting data. By mastering its basic syntax, field separators, pattern matching, arithmetic capabilities, built‑in variables, custom functions, and control structures, you can efficiently handle a wide range of text‑processing tasks.