Master awk: Powerful Text Processing Tricks for Linux Users

What is awk?

If you work with Linux a lot, learning awk is worthwhile. awk is an extremely powerful text analysis tool that reads files line by line, splits each line by a specified delimiter, and allows various analysis and processing.

With awk you can create programs to read input files, sort data, process data, perform calculations, generate reports, and many other functions.

awk usage examples

Below are simple examples to get acquainted with awk.

Example 1: Print filenames

ll | awk '{print $9}'

This command displays only the list of filenames. awk receives the output of ll, processes each line, splits by spaces (default delimiter), and prints the ninth field.

Example 2: Show username and home directory

awk -F ':' '{print $1"\t"$6}' /etc/passwd

The command processes /etc/passwd, which uses ':' as a field separator, and prints the first and sixth columns (username and home directory).

Example 3: Filter lines containing “root”

awk -F ':' '/root/{print $0}' /etc/passwd

Using a pattern, awk prints only lines that match “root”. Regular expressions can be used, e.g., /^root/ to match lines starting with root.

Example 4: Print file metadata

awk -F ':' '{print "filename:" FILENAME ",linenumber:" NR ",columns:" NF ",linecontent:"$0}' /etc/passwd

awk provides built‑in variables such as FILENAME (file name), NR (record number), and NF (number of fields).

Example 5: BEGIN and END blocks

awk -F ':' 'BEGIN {print "header"} {print $1} END {print "foot"}' /etc/passwd

BEGIN runs before processing, END runs after processing.

Example 6: Calculate total size of files in a directory

ls -l | awk 'BEGIN {size=0;} {size=size+$5;} END{print size/1024/1024,"M"}'

This command sums the sizes (field 5) of all files and prints the total in megabytes. It demonstrates variable definition and accumulation.

Example 7: Exclude directories from size calculation

ls -l | awk 'BEGIN {size=0;} {if($5!=4096){size=size+$5;}} END{print size/1024/1024,"M"}'

Here a conditional statement filters out entries where the size equals 4096 (typical directory size). awk’s conditionals are similar to C (if, while, for, break, continue).

Through these examples you can see awk’s capabilities: built‑in variables, custom variables, conditional statements, regular expressions, and more, making it effectively a programming language for text processing.

awk also has many advanced uses; interested readers can explore further.