Turbocharge Linux Commands with GNU Parallel: Faster grep, awk, bzip2, and More

How to Use Multi‑Core CPUs to Accelerate Linux Commands (awk, sed, bzip2, grep, wc, etc.)

Many common Linux utilities such as grep , bzip2 , wc , awk , and sed are single‑threaded and only use one CPU core, even on a multi‑core machine.

GNU Parallel can distribute the workload across all cores by using the --pipe (or --spreadstdin) option, effectively turning these commands into a map‑reduce pipeline.

BZIP2

Traditional compression:

cat bigfile.bin | bzip2 --best > compressedfile.bz2

Parallel version:

cat bigfile.bin | parallel --pipe --recend '' -k bzip2 --best > compressedfile.bz2

GREP

Traditional search: grep pattern bigfile.txt Parallel search examples:

cat bigfile.txt | parallel --pipe grep 'pattern'

cat bigfile.txt | parallel --block 10M --pipe grep 'pattern'

The --block 10M option tells each core to process roughly ten million lines, allowing you to tune the chunk size.

AWK

Summing a large numeric file: cat rands20M.txt | awk '{s+=$1} END {print s}' Parallel version:

cat rands20M.txt | parallel --pipe awk '{s+=$1} END {print s}' | awk '{s+=$1} END {print s}'

The --pipe flag splits the input and feeds each chunk to separate awk processes, then aggregates the results.

WC

Traditional line count: wc -l bigfile.txt Parallel line count:

cat bigfile.txt | parallel --pipe wc -l | awk '{s+=$1} END {print s}'

This runs wc -l in parallel on many chunks and then sums the partial counts with awk.

SED

Traditional substitution: sed s^old^new^g bigfile.txt Parallel substitution:

cat bigfile.txt | parallel --pipe sed s^old^new^g

You can pipe the output to a file if you need to store the modified content.