Practical grep, awk, and sed use cases for daily operations

1. grep: Find problem lines in logs

1.1 Real‑world scenario

The simplest case is extracting all lines that contain ERROR from an application log, e.g. /var/log/app/app.log: grep "ERROR" /var/log/app/app.log In practice, administrators often need more advanced queries:

Limit results to a specific time window.

Show surrounding context (e.g., 5 lines before and after the match).

Count occurrences of a particular error.

Exclude known, irrelevant messages.

Recursively search multiple files.

1.2 Core principle and common options

grep reads input line‑by‑line and prints a line when the regular expression matches. It cannot match across lines; use -z or a different tool for multi‑line patterns. -i: ignore case. -v: invert match (exclude matching lines). -n: show line numbers. -c: only count matching lines. -r or -R: recursive directory search. -l: list only file names that contain a match. -A n: show n lines after a match. -B n: show n lines before a match. -C n: show n lines of context (both before and after). -w: match whole words only. -o: output only the matching part.

Example showing context for an OutOfMemoryError:

grep -C 5 "OutOfMemoryError" /var/log/app/app.log

Sample output demonstrates how surrounding lines reveal the root cause:

2026-07-14 09:12:01 INFO  Processing batch job id=8821
2026-07-14 09:12:03 INFO  Loaded 500000 records into memory
2026-07-14 09:12:05 WARN  Heap usage at 92%
2026-07-14 09:12:06 ERROR OutOfMemoryError: Java heap space
2026-07-14 09:12:06 ERROR Thread [batch-worker-3] terminated
2026-07-14 09:12:07 INFO  GC triggered, heap usage dropped to 45%

Counting a specific error:

grep -c "Connection refused" /var/log/app/app.log

Excluding known noise:

grep "ERROR" /var/log/app/app.log | grep -v "DeprecationWarning"

Recursive search for a configuration key:

grep -rn "max_connections" /etc/mysql/

1.3 Validation experiments – regex pitfalls

grep uses Basic Regular Expressions (BRE) by default. To enable extended regex (ERE), add -E. For example, the pipe character | does **not** act as logical OR in BRE:

echo "connection timeout" | grep "timeout|refused"

The above matches because the literal string contains timeout, not because | works as OR. The correct form is:

echo "connection timeout" | grep -E "timeout|refused"

Another common mistake is forgetting to escape the dot in an IP address:

grep "192.168.1.1" access.log   # matches 192a168x1y1 as well
grep "192\.168\.1\.1" access.log   # correct

1.4 Common pitfalls

Assuming grep can match across lines – it cannot without -z or another tool.

Chaining multiple greps (e.g., grep A file | grep B | grep C) is slower than a single extended regex like grep -E "A.*B.*C" or using awk for complex logic.

Neglecting -w leads to substring matches (e.g., searching for 80 also matches 8080).

2. sed: Stream‑editing for in‑place modifications

2.1 Real‑world scenario

sed is frequently used to batch‑replace values in configuration files, such as swapping a test domain for a production domain in an Nginx config.

2.2 Core principle

sed reads input line‑by‑line, applies editing commands, and writes the result to standard output. By default it does **not** modify the original file.

Basic substitution: sed 's/old_content/new_content/' filename Adding the g flag makes the substitution global on each line: sed 's/old_content/new_content/g' filename Preview changes before committing:

sed 's/test.example.com/prod.example.com/g' nginx.conf

Apply changes in‑place with backup:

cp nginx.conf nginx.conf.bak.$(date +%Y%m%d%H%M%S)
sed -i 's/test.example.com/prod.example.com/g' nginx.conf

GNU sed also supports -i.bak to create a backup automatically:

sed -i.bak 's/test.example.com/prod.example.com/g' nginx.conf

2.3 Common composition patterns

Delete empty lines: sed '/^$/d' filename Delete comment lines starting with #: sed '/^#/d' filename Print a specific line range (e.g., lines 10‑20): sed -n '10,20p' filename Replace within a section delimited by patterns:

sed '/\[section_a\]/,/^\[/ s/enabled=false/enabled=true/' config.ini

Chain multiple substitutions:

sed -e 's/foo/bar/g' -e 's/baz/qux/g' filename

2.4 Common pitfalls

Running sed -i on production files without a backup can permanently corrupt data.

Using greedy regex like s/<.*>//g to strip HTML tags may delete everything between the first < and the last >. POSIX regex lacks non‑greedy quantifiers; consider awk or perl for such tasks.

Attempting column‑oriented processing with sed is fragile; use awk for field‑based logic.

3. awk: Field‑based statistics and calculations

3.1 Real‑world scenario

awk excels at processing structured text such as space‑ or delimiter‑separated logs, ps output, or df output. Typical tasks include counting requests per IP, summing numeric columns, or filtering rows based on conditions.

3.2 Core principle

awk splits each input line into fields ( $1, $2, …, $NF) using whitespace by default (or a custom -F separator). The processing model is pattern {action}. If the pattern is omitted, the action runs on every line.

Extract the first column: awk '{print $1}' access.log Count occurrences of each IP (assuming the IP is the first column):

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

Note that awk only extracts; sort and uniq perform the aggregation.

3.3 Common composition patterns

Parse /etc/passwd (colon‑separated) and print username and shell: awk -F: '{print $1, $NF}' /etc/passwd Filter df -h output for usage > 80 %:

df -h | awk 'NR>1 {gsub("%","",$5); if ($5+0 > 80) print $0}'

Sum a numeric column (e.g., column 7 contains response time in ms) and compute average:

awk '{sum+=$7; count++} END {print "Total requests:", count, "Avg latency(ms):", sum/count}' app_access.log

Use BEGIN and END blocks for headers and final reports:

awk 'BEGIN {print "IP statistics"; print "----------"} {count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log

3.4 Validation experiments – field separator nuances

Default separator treats any run of spaces/tabs as one delimiter. Explicitly setting -F" " (single space) splits on **each** space, which can produce empty fields when the input contains multiple spaces.

echo "a    b   c" | awk '{print $2}'          # outputs: b
echo "a    b   c" | awk -F" " '{print $2}' # outputs: (empty)

Therefore, for commands like ps, df, or free that align columns with variable spacing, rely on the default separator or use a regex separator such as -F"[ ]+".

3.5 Common pitfalls

Using a fixed -F" " on output with variable spacing leads to mis‑aligned fields.

Confusing string comparison ( == with quoted strings) with numeric comparison can cause always‑false tests.

Assuming a field number is stable; when the number of columns varies, prefer $NF or check NF first.

4. Combining grep, sed, and awk in real troubleshooting

A typical end‑to‑end workflow extracts 5xx responses from an Nginx access log, limits the analysis to the current hour, counts the most frequent URLs, and shows the top ten:

CURRENT_HOUR=$(date '+%Y-%m-%d %H')
grep "^${CURRENT_HOUR}" access.log \
  | awk '$9 ~ /^5[0-9][0-9]$/ {print $7}' \
  | sort | uniq -c | sort -rn | head -10

Key points:

Use grep first to narrow the dataset (time filter) – this reduces the amount of data handed to awk. awk matches the status code column (here column 9) and outputs the request path (column 7).

Standard sort | uniq -c | sort -rn chain provides counting and ranking.

Always verify column numbers against the actual log_format definition in the Nginx configuration before relying on hard‑coded indices.

5. Engineering recommendations

Before applying sed -i to production files, test the regex on a copy or in a non‑production environment and always keep a backup.

For large log files (several GB), pre‑filter with grep to reduce the data volume before invoking more expensive awk scripts.

When a troubleshooting pipeline becomes complex, store the commands in a commented shell script instead of typing them ad‑hoc; this improves readability and reduces the risk of accidental destructive operations.

Always confirm field positions and delimiters against the actual log or command output (e.g., nginx -T | grep -A5 "log_format") before writing awk one‑liners.

Leverage the strengths of each tool – grep for fast line selection, sed for in‑place text transformation, and awk for column‑wise computation – rather than forcing a single tool to do everything.