Mastering grep, sed, awk: The Swiss Army Knife for Log Processing

Problem Background

Half of production troubleshooting time is spent reading logs, and half of that time uses grep/sed/awk. Despite their apparent simplicity, these tools cause frequent production incidents:

Greedy regex in grep -o 'GET /api/.*' matches entire lines instead of URLs, corrupting Top10 statistics. sed -i without backup suffix destroys config files when regex is wrong, with no rollback.

BSD sed -i on macOS requires an argument, unlike GNU sed, breaking cross-platform scripts.

Awk loading multi-GB logs into associative arrays causes OOM kills.

Windows CRLF line endings leave invisible \r in CSV fields, breaking awk field splitting.

Missing --line-buffered on grep delays real-time tail -F output by minutes.

Sensitive data (phone numbers, IDs) leaked when grep output shared without masking.

Core difficulties: regex greediness, BRE/ERE/PCRE differences, GNU vs BSD implementation gaps, large-file memory/buffering, encoding/line-ending issues, and mandatory data masking in production.

Tool Roles

grep : line-level filtering — selects matching lines, does not modify content. Strength: speed, simplicity.

sed : stream editor — reads line by line, applies edits (substitute, delete, insert, print). Strength: replacement and cleansing.

awk : field processing — splits by delimiter, computes/aggregates on fields. Strength: columnar calculation and associative arrays.

One-liner summary: grep picks lines, sed edits lines, awk computes columns.

Regex Flavors

BRE (default): grep 'a.b', sed 's/a/b/'. Metacharacters +, ?, {n,m}, (), | are literal; escape as \+, \?, \{n,m\}, \(, \), \|.

ERE ( grep -E, sed -E / -r, awk default): +, ?, {n,m}, (), | are metacharacters directly.

PCRE ( grep -P, GNU only): supports non-greedy .*?, lookahead (?=), named capture. Not portable (BSD grep lacks -P).

Guidance: use BRE/ERE for simple patterns; only use -P when non-greedy/lookahead needed and GNU grep confirmed; prefer sed -E for clearer substitutions.

Field Separators & Variables

Awk default FS: contiguous whitespace (spaces/tabs). awk -F',' for comma, -F'\t' for tab, -F'|' for pipe.

Multi-char FS: awk -F'\|\|' or awk -v FS='::'. OFS sets output separator; print $1,$2 joins with OFS (default space).

Key variables: $0 (whole line), $1..$NF (fields), NR (global line number), FNR (per-file line number), NF (field count), FILENAME, FS/OFS/RS/ORS.

Buffering & Real-time

grep/sed/awk default to block buffering when output redirected/piped.

Real-time requires grep --line-buffered, sed -u, awk fflush(). tail -F is line-buffered by default, ideal for live tracking.

High-Risk Actions

sed -i

without backup: irreversible corruption. sed -i with wrong regex: mass mis-replacement.

Awk loading huge files into arrays: OOM.

Grep outputting sensitive data: leaks.

sed/awk unescaped special chars in input: injection/mismatch.

End-to-End Log Troubleshooting Loop

Locate log files, size, compression status.

Grep coarse filter (keyword, status, time, URL).

Sed cleanse (remove fields, normalize format, mask).

Awk aggregate (TopN, UV, error rate, latency distribution).

Cross-correlate multiple metrics for root cause.

Output to permission-controlled dir, mask before sharing.

Practical Steps & Commands

5.1 Pre-check: Locate & Quantify Logs

# Find logs
ls -lh /var/log/nginx/
find /data/log -name 'access*.log' -printf '%s\t%p
' | sort -rn | head
# Line count (wc -l slow on huge files; sample first)
wc -l /var/log/nginx/access.log
# Preview format
head -n 3 /var/log/nginx/access.log
# Check compression
ls -lh /var/log/nginx/access.log*.gz
file /var/log/nginx/access.log.1.gz

Judgment: sample first 100k lines ( head -n 100000) before full run; use zgrep / zcat for compressed logs; confirm field positions for awk -F.

5.2-5.4 Grep Essentials, Regex, Large Files & Real-time

Key flags: -i (case-insensitive), -n (line numbers), -v (invert), -c (count matching lines, not occurrences), -o (only matched part), -l (filenames with matches), -r (recursive), -E (ERE), -P (PCRE, GNU), -F (fixed string), -f (pattern file), -A/-B/-C (context), --line-buffered (real-time), zgrep / zcat for .gz.

Regex traps: BRE requires \{3\} vs ERE {3}; IP regex ([0-9]{1,3}\.){3}[0-9]{1,3} matches 999.999.999.999 (good enough for coarse filtering); grep -P non-greedy faster on long lines but GNU-only.

Real-time pattern: tail -F /var/log/app.log | grep --line-buffered 'ERROR'; for huge logs, tail -n 100000 | grep ... limits scan range.

5.5-5.8 Sed Basics, In-place Editing, CRLF/BOM, Advanced

Substitution flags: g (global), N (Nth match), p (print), i (ignore case, GNU). -n suppresses default output; combine with p to print only matches. & = whole match; \1..\9 = capture groups.

In-place editing (critical) : GNU sed -i.bak 's/old/new/g' file creates file.bak; BSD/macOS requires sed -i '.bak' ... or sed -i '' ... for no backup. Production rule: always sed -i.bak, then diff verify before dropping backup.

CRLF/BOM handling: sed -i 's/\r$//' file.csv strips \r; sed -i '1s/^\xEF\xBB\xBF//' file.csv removes UTF-8 BOM; cat -A shows ^M$ for CRLF.

Advanced sed (address ranges, multi-line): sed -n '/BEGIN/,/END/p', sed 'N;s/\n/ /', sed ':a;N;$!ba;s/\n/ /g' (slurps whole file — memory risk on large files). Prefer awk for complex multi-line logic.

5.9-5.13 Awk Basics, BEGIN/END, PV/UV/Status, Multi-file, Large Files

Field access: awk '{print $1, $7}', awk -F',' '{print $1,$3}', $NF last field, $(NF-1) penultimate. Conditions as patterns: awk '$3 > 100 {print}', awk '$1 == "ERROR"', awk '/ERROR/'.

Aggregation: END{print NR} (line count), {sum+=$3} END{print sum}, {sum+=$3; n++} END{print sum/n}, associative arrays count[$9]++ for status distribution. TopN:

awk '{count[$9]++} END{for(s in count) print count[s], s}' | sort -rn | head

.

Multi-file: FILENAME and FNR==1 detect boundaries; NR==FNR{...; next} pattern builds lookup from first file.

Large-file safety: UV (dedup) must use sort | uniq -c instead of awk arrays; sums/counts use scalar variables (safe); sampling awk 'NR%100==0' for trend estimation.

5.14 Data Masking

# Phone: keep first 3, last 4
sed -E 's/(1[3-9][0-9])[0-9]{4}([0-9]{4})/\1****\2/g' app.log
# ID card: keep first 6, last 4
sed -E 's/([0-9]{6})[0-9]{8}([0-9]{4})/\1********\2/g' app.log
# IP: keep first two octets
sed -E 's/([0-9]+\.[0-9]+)\.[0-9]+\.[0-9]+/\1.*.*/g' app.log
# Awk mask + aggregate
awk '{ip=$1; split(ip,a,"."); masked=a[1]"."a[2]".*.*"; cnt[masked]++} END{for(m in cnt) print cnt[m], m}' access.log | sort -rn | head

Rules: output to /data/secure/ (700 perms); regex must cover all formats; periodic scan grep -rE '1[3-9][0-9]{9}' /data/secure/.

Complete Troubleshooting Case: Nginx 5xx Spike

Symptom : 14:50 business reports errors; monitoring shows 5xx from 0.1% to 15% (14:45-14:55).

Commands :

# 5xx time distribution
awk '$9 ~ /^5/ {print $4}' access.log | sort | uniq -c | sort -rn | head
# 5xx URLs
awk '$9 ~ /^5/ {split($7,a," "); print a[2]}' access.log | sort | uniq -c | sort -rn | head -10
# Upstream errors (if logged)
grep -E ' 5[0-9]{2} ' access.log | grep -oE 'upstream:[^ ]+' | sort | uniq -c | sort -rn
# Error log in window
awk '$4 ~ /14:[45][0-9]:/' error.log | grep -E 'upstream|connect|timeout|refused'

Root cause : error.log shows

connect() failed (111: Connection refused) while connecting to upstream

; app server 10.0.0.21 process died at 14:46. dmesg reveals Killed process 12345 (java) total-vm:... — OOM due to memory leak/insufficient heap.

Fix : restart app (priority), verify 5xx drop, then address root cause (increase heap, fix leak). Rollback : if restart fails, revert to healthy instance, rollback recent deploy, or temporarily remove upstream from nginx.

Postmortem improvements : OOM alert at >80% memory; systemd Restart=always; nginx upstream health checks; retain error.log ≥7 days.

Case: Field Misalignment in Awk

Symptom : awk '{code[$9]++}' yields garbage, not 200/404/500.

Cause : combined log format $request = "GET /api/orders HTTP/1.1" contains spaces, shifting field positions.

Fixes : (1) Change log_format to pipe-delimited (recommended); (2) Awk regex extraction match($0, /" ([0-9]{3}) /, m){code[m[1]]++}; (3) gawk FPAT: awk 'BEGIN{FPAT="([^ ]+)|(\"[^\"]+\")"} {code[$8]++}' (defines field pattern, not separator).

Case: Sed -i Batch Config Corruption

Symptom : sed -i 's/port=3306/port=3307/g' /etc/app/*.conf used greedy s/port=.*/port=3307/, truncating lines; no backup.

Recovery : restore from git/etcd, rsync from peer rsync -av 10.0.0.22:/etc/app/ /etc/app/, or re-apply Ansible playbook.

Prevention : always sed -i.bak; dry-run sed 's/...' file | grep '^port='; precise regex sed -i.bak -E 's/^port=[0-9]+/port=3307/'; diff verify.

Risk Checklist

Operation : sed -i no backup — Risk : Irreversible — Must Do : Enforce sed -i.bak Operation : sed -i greedy regex — Risk : Over-replace — Must Do : Precise match + dry-run

Operation : Awk huge array — Risk : OOM — Must Do : Use sort/uniq or sampling

Operation : Grep outputs PII — Risk : Leak — Must Do : Mask before output

Operation : grep -P cross-platform — Risk : Incompatible — Must Do : Confirm GNU or use ERE

Operation : CRLF unhandled — Risk : Field shift — Must Do : Preprocess sed 's/\r$//' Operation : BOM present — Risk : First field corrupted — Must Do :

sed '1s/^\xEF\xBB\xBF//'

Regex & Performance Traps

BRE vs ERE escaping differences (most common error source).

Greedy .* matches to line end; use .*? (PCRE) or rewrite. . doesn't match newline by default. ^ / $ anchors; ^ inside [^abc] negates. grep -F for literal strings containing regex metacharacters.

Grep large file without --line-buffered in pipeline: latency.

Awk length(arr) O(n) — avoid in loops.

Sed -i uses temp disk — disk full = failure.

Sort large files: add --buffer-size or shard.

Multiple greps on same file slower than single awk with multiple conditions.

Verification Methods

Semantics: test pipeline on head -n 1000 sample first.

Sed replacement: dry-run sed 's/old/new/g' file | grep 'new', then diff file file.bak.

Masking: scan output grep -rE '1[3-9][0-9]{9}|[0-9]{17}[0-9Xx]' /data/secure/ — zero output = clean.

Stats accuracy: cross-validate UV via awk '{ip[$1]++} END{print length(ip)}' vs awk '{print $1}' | sort -u | wc -l.

Rollback Strategies

Sed -i: mv file.bak file or restore from git/peer/Ansible.

Script output: write to output.new, verify, then mv output.new output.final.

Alert daemon: systemctl stop/disable log_alert, remove unit file, daemon-reload.

Production Discipline (12 Rules)

sed -i

mandatory backup + dry-run.

Large files: prefer sort/uniq over awk arrays, or sample.

Mask sensitive data before any output.

Cross-platform scripts: distinguish GNU/BSD ( sed -i, grep -P).

Preprocess CRLF/BOM: sed 's/\r$//'.

Custom log_format with delimiters (e.g., |) for awk friendliness.

Real-time tracking: --line-buffered / -u / fflush.

Daemon scripts under systemd with Restart=always.

Metrics via node_exporter textfile; thresholds tuned to business baselines.

Log retention ≥7 days, compressed, permission-controlled.

Appendices Highlights

A-G : grep/sed/awk cheat sheets with 50+ recipes (multi-condition AND, IP extraction, time-range slicing, config editing, CSV column swap, deduplication, percentile calc, join-like merge, etc.).

E : Large file sharding + parallel awk ( split -l 5000000, xargs -P 8, merge results).

F : BSD vs GNU diff table (sed -i, -E/-r, grep -P, awk extensions, sort -R, etc.); macOS advice: brew install gsed gawk ggrep.

G : Tool selection matrix — grep for simple filter, awk for column stats, sort/uniq for huge dedup, tail+grep for real-time.

H : Common error diagnostics (sed stdin -i, BSD -i arg, mawk vs gawk, locale char-class, field shift, OOM, sort disk full).

I : Automated nginx log report script (Top10 IP/URL, 5xx, large responses, slow requests) with masked output to /data/report.

J : 15-item pre-flight checklist (backup, precise regex, no full arrays, masking, GNU/BSD, CRLF/BOM, field confirm, line-buffered, systemd, temp cleanup, secure output, sample validation, cross-check, baseline thresholds, retention).

K : One-liner card for quick reference (status dist, Top10 IP, 5xx rate, live ERROR tail, phone mask, CRLF strip, config change with backup, dedup, time-range, big-file sum, zgrep).

L : Logrotate integration — zcat *.gz | awk ..., per-file loop to avoid mid-run failure, delaycompress.

M : journalctl + three swords — --since/--until time indexing, --no-pager, -o json | jq, persistent storage config.

N : Smart alert script with masking, dedup window (5 min), state-file rate limiting, systemd hardening ( ProtectSystem=strict, ReadWritePaths).

O : Five-stage learning path (query/stats → regex/pipes → cleansing/masking → big data/performance → automation/monitoring) with hands-on exercises per stage.

P : Postmortem template capturing timeline, commands used, root cause, fixes, preventive actions.

Q : Five golden rules — sample first, backup before write, prevent OOM, mask early, closed-loop troubleshooting (phenomenon → filter → cleanse → aggregate → root cause → verify).