Mastering awk: Essential Commands, Scripts, and Advanced Techniques

awk Command Syntax

awk [options] 'script' var=value file(s)
awk [options] -f scriptfile var=value file(s)

    -F fs   # set input field separator (string or regex)
    -v var=value   # assign a user‑defined variable
    -f scriptfile   # read awk program from a file

The script consists of patterns and actions.

Pattern Types

Regular expression patterns

Relational expressions (string or numeric comparisons)

Pattern‑matching expressions using ~ (match) and !~ (no match)

Special blocks: BEGIN, pattern,

END

awk Script Basic Format

awk 'BEGIN{ commands } pattern{ commands } END{ commands }' file

# Example:
awk 'BEGIN{ i=0 } { i++ } END{ print i }' filename

Execution Process

Step 1 – BEGIN block: Executed before any input is read; useful for initializing variables or printing headers.

Step 2 – Pattern block: For each input line, awk evaluates the pattern; if it matches, the associated commands run. If no pattern is provided, the default action is { print }, printing each line.

Step 3 – END block: Executed after all input has been processed; typically used for summarizing results.

Built‑in Variables

$n        # nth field of the current record
$0        # entire current line
ARGC      # number of command‑line arguments
ARGIND    # index of the current file
ARGV      # array of command‑line arguments
CONVFMT   # numeric conversion format (default "%.6g")
ENVIRON   # associative array of environment variables
ERRNO     # description of the last system error
FIELDWIDTHS # list of field widths
FILENAME  # name of the current input file
NR        # total record number (line number across all files)
FNR       # record number within the current file
FS        # input field separator (default whitespace)
IGNORECASE# if true, pattern matching ignores case
NF        # number of fields in the current record
OFMT      # output format for numbers (default "%.6g")
OFS       # output field separator (default space)
ORS       # output record separator (default newline)
RS        # input record separator (default newline)
RSTART    # start index of the last match
RLENGTH   # length of the last match
SUBSEP    # subscript separator for multidimensional arrays

Passing External Variables

Use -v to assign a variable from the shell: VAR=10000; echo | awk -v VARIABLE=$VAR '{ print VARIABLE }' Assign variables after the script:

var1="aaa" var2="bbb"; echo | awk '{ print v1,v2 }' v1=$var1 v2=$var2

When reading from a file:

awk '{ print v1,v2 }' v1=$var1 v2=$var2 filename

Operations

Arithmetic: +, -, *, /, %, ++, -- Assignment: =, +=, -=, *=, /=, %=, **= Logical: ||, && Relational: <, <=, >, >=, !=, == Regex: ~ (match) and !~ (no match)

awk 'BEGIN{a="100testa"; if(a ~ /^100*/){print "ok"}}'
ok

Advanced Input/Output

next – skip the current record and continue with the next one.

awk 'NR%2==1{next}{print NR,$0}' text.txt
# Lines with an odd line number are skipped.

getline – read a line from a file, pipe, or standard input into a variable.

It returns 1 on success, 0 on EOF, and –1 on error.

File Operations

Open a file: open("filename") Close a file: close("filename") Redirect output:

echo | awk '{printf("hello world
") > "datafile"}'

Loop Structures

for loop

for (var in array) { statements }
for (init; condition; increment) { statements }

while loop

while (condition) { statements }

do…while loop

do { statements } while (condition)

Other control statements

break      # exit the current loop
continue   # skip to the next iteration
next       # read the next input line
exit       # terminate the script (runs END block first)

Examples

# Print each line of /etc/passwd
awk '{print}' /etc/passwd

# Print a blank line for each input line
awk '{print " "}' /etc/passwd

# Print the first field using ':' as separator
awk -F":" '{print $1}' /etc/passwd

# Count blank lines in a file
awk 'BEGIN{X=0} /^$/{X+=1} END{print "I find",X,"blank lines."}' test

# Sum file sizes (excluding directories)
ls -l | awk 'BEGIN{sum=0} !/^d/{sum+=$5} END{print "total size is",sum}'

For a full list of examples, see the original source.