Master AWK: Essential Text‑Processing Commands with Real‑World Examples

Text Analysis

AWK can quickly analyze text files, such as counting lines, words, or characters.

# Count lines in a file
awk 'END { print NR }' filename

# Count words in a file
awk '{ total += NF } END { print total }' filename

# Count characters in a file
awk '{ total += length } END { print total }' filename

In these examples, NR is the current line number, NF is the number of fields (words) on the line, and length returns the character count of the line. The END block prints the final result.

Text Formatting

Beyond analysis, AWK can reformat output, reorder fields, add delimiters, or insert header lines.

# Reorder fields and add a header line
awk 'BEGIN { print "Name\tAge\tGender" } { print $2, $3, $1 }' filename

# Use a comma as the output field separator
awk -v OFS=',' '{ print $1, $2 }' filename

Here $1, $2, etc., refer to individual fields, OFS sets the output separator, and the BEGIN block prints a header before processing the data.

Pattern Matching

AWK supports regular‑expression pattern matching, enabling flexible text filtering.

# Print lines containing the word "error"
awk '/error/ { print }' filename

# Print lines where the first field starts with "A"
awk '$1 ~ /^A/ { print }' filename

The pattern /error/ matches any line containing the word "error", while $1 ~ /^A/ matches lines whose first field begins with "A". The print statement outputs the matching lines.

Computation

AWK can also perform arithmetic operations, such as summing a column or calculating averages.

# Sum the values in the second column
awk '{ sum += $2 } END { print sum }' filename

# Compute the average of the third column
awk '{ total += $3 } END { print total/NR }' filename

Variables sum and total accumulate values, and the END block outputs the final computation.

Custom Functions

Users can define their own functions for reusable logic.

# Define a function to calculate the square of a number
awk 'function square(x) { return x*x } { print square($1) }' filename

The function square returns the square of its argument, allowing the script to process each line’s first field accordingly.

Flow Control

AWK includes standard control structures such as if‑else and for loops.

# Conditional output based on the first field
awk '{ if ($1 > 50) print "Pass"; else print "Fail" }' filename

# Generate a multiplication table
awk 'BEGIN { for (i=1; i<=10; i++) { for (j=1; j<=10; j++) { printf "%d\t", i*j } print "" } }'

The if‑else statement chooses output based on a condition, while nested for loops produce a formatted multiplication table.

Counting Unique IP Addresses in a Log File

As a practical example, AWK can tally how many times each IP address appears in an access log.

# Count occurrences of each IP address
awk '{ count[$1]++ } END { for (ip in count) print ip, count[ip] }' access.log

The associative array count stores the frequency of each IP (the first field). After processing the file, the END block iterates over the array and prints each IP with its count.

Conclusion

AWK is a versatile and powerful tool for text processing on Unix and Linux systems. By mastering its basic commands for analysis, formatting, pattern matching, arithmetic, custom functions, and flow control, you can efficiently handle a wide range of data‑manipulation tasks.