How to Process 10 GB of Logs in 30 Seconds with grep, sed, and awk

Overview

During a Double‑Eleven incident the author was woken up by a 12 GB Nginx access log and needed to locate the root cause within minutes. Traditional editors and Python scripts were too slow, so a one‑liner composed of grep , sed and awk identified a malicious crawler IP in under 30 seconds, proving the lasting value of the "Shell three musketeers" for rapid SRE troubleshooting.

Technical Characteristics

The three tools share a stream‑processing model: they read one line at a time, process it, and immediately release memory, making the memory footprint independent of file size. They are written in C, compiled for decades, and call the OS I/O directly, which explains why awk can be 5‑10× faster than a naïve Python readlines() approach.

grep uses highly optimized DFA engines; the -F (fixed‑string) mode and GNU grep version 3.8+ are especially fast. sed excels at in‑place text substitution, while awk is a full programming language capable of field‑wise calculations, associative arrays, and BEGIN/END blocks.

Tool Roles (Three‑Musketeer Division of Labor)

grep – fast pattern filtering, e.g., grep -n "ERROR" access.log sed – stream editing, e.g.,

sed -i.bak 's/worker_processes auto/worker_processes 8/' /etc/nginx/nginx.conf

awk – complex field processing, e.g.,

awk '{ip[$1]++} END{for(i in ip) print ip[i], i}' access.log

Applicable Scenarios

Typical use cases include fault isolation, log statistics, configuration bulk updates, CSV/JSON data cleaning, and real‑time monitoring with tail. Scenarios that do not fit are multi‑file correlation (better handled by ELK), persistent storage needs, and deeply nested data structures.

Environment Requirements

The commands were tested on Ubuntu 22.04 LTS, CentOS 8, and macOS Sonoma with GNU grep 3.8+, GNU sed 4.8+, GNU awk (gawk) 5.1+, and optionally ripgrep 14.0+.

On macOS, install the GNU versions via Homebrew: brew install grep sed gawk ripgrep After installation, the GNU binaries are prefixed with g (e.g., ggrep, gsed, gawk) or can be aliased in .bashrc:

alias grep='ggrep'
alias sed='gsed'
alias awk='gawk'

Detailed Steps

Preparation

A helper script generates a synthetic 1 GB Nginx log ( access.log) with random IPs, URLs, status codes, user‑agents and response times. The script demonstrates Bash loops, array indexing, and awk for random response time generation.

#!/bin/bash
# generate_nginx_log.sh – generate simulated Nginx logs
LOG_FILE="access.log"
TOTAL_LINES=10000000  # ~1 GB
IPS=("192.168.1.100" "192.168.1.101" "10.0.0.50" "10.0.0.51" "172.16.0.10" "8.8.8.8" "1.1.1.1" "203.0.113.50" "198.51.100.23" "185.220.101.42")
URLS=("/api/users" "/api/orders" "/api/products" "/api/search" "/static/js/main.js" "/static/css/style.css" "/images/logo.png" "/api/payment" "/health" "/metrics" "/api/v2/data")
STATUS_CODES=("200" "200" "200" "200" "200" "201" "301" "302" "400" "401" "403" "404" "500" "502" "503")
USER_AGENTS=("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X)" "curl/7.88.1" "python-requests/2.28.0" "Googlebot/2.1 (+http://www.google.com/bot.html)")

echo "开始生成日志文件，共 $TOTAL_LINES 行..."
for ((i=1; i<=TOTAL_LINES; i++)); do
  ip=${IPS[$RANDOM % ${#IPS[@]}]}
  url=${URLS[$RANDOM % ${#URLS[@]}]}
  status=${STATUS_CODES[$RANDOM % ${#STATUS_CODES[@]}]}
  ua=${USER_AGENTS[$RANDOM % ${#USER_AGENTS[@]}]}
  size=$((RANDOM % 50000 + 100))
  response_time=$(awk -v min=0.001 -v max=5.0 'BEGIN{srand(); print min+rand()*(max-min)}')
  timestamp=$(date "+%d/%b/%Y:%H:%M:%S +0800")
  echo "$ip - - [$timestamp] \"GET $url HTTP/1.1\" $status $size \"$ua\" $response_time"
done > "$LOG_FILE"

echo "日志生成完成：$LOG_FILE，大小：$(du -h $LOG_FILE | cut -f1)"

Core Configuration

grep frequently used options (ordered by usefulness):

# Search with line numbers
grep -n "ERROR" file
# Case‑insensitive search
grep -i "error" file
# Count matches only
grep -c "ERROR" file
# Invert match
grep -v "DEBUG" file
# Show context lines
grep -A 5 "Exception" file
# Recursive search
grep -r "TODO" ./src
# Whole‑word match
grep -w "error" file
# Output only the matching part
grep -o "ip=[0-9.]*" file
# Extended regex
grep -E "err|warn" file
# Perl regex (PCRE)
grep -P "\d{4}" file
# Fixed‑string (fast)
grep -F "fixed_string" file

sed essential patterns:

# Simple substitution
sed 's/old/new/' file
# Global substitution, case‑insensitive
sed 's/old/new/gi' file
# In‑place edit with backup (GNU)
sed -i.bak 's/worker_processes auto/worker_processes 8/' /etc/nginx/nginx.conf
# Delete lines containing DEBUG
sed '/DEBUG/d' file
# Print only matching lines
sed -n '/ERROR/p' file
# Range delete (lines 10‑20)
sed '10,20d' file
# Insert after a pattern
sed '/EOF/a # End of file' file

awk core idioms (field‑wise processing, associative arrays, BEGIN/END blocks):

# Print first field (IP)
awk '{print $1}' access.log
# Count occurrences of each IP
awk '{ip[$1]++} END{for(i in ip) print ip[i], i}' access.log
# Filter rows where 3rd field > 100
awk '$3 > 100 {print}' access.log
# Compute average response time per URL (last field is time)
awk '{url=$7; time=$NF; sum[url]+=time; cnt[url]++} END{for(u in cnt) printf "%s %.3f %d
", u, sum[u]/cnt[u], cnt[u]}' access.log | sort -k2 -rn
# BEGIN block to set field separator
awk -F":" '{print $1}' file
# Format output columns
awk '{printf "%-15s %10d
", $1, $2}' file

Performance Comparison

Processing a 1 GB log (≈10 M lines) with three different approaches yielded:

Method 1 (grep → sort → uniq → sort): 45 s, 2 GB temporary files.

Method 2 (pure awk counting): 28 s, 800 MB peak memory.

Method 3 (awk with time filter then sort): 15 s, 400 MB memory.

The speed advantage of Method 2 stems from awk’s in‑memory associative array, avoiding the heavy disk I/O of sort.

Best‑Practice Recommendations

Filter with grep before handing data to awk to reduce processed volume.

Use LC_ALL=C for pure ASCII data to gain 2‑3× speed.

For very large files, split them and process in parallel with GNU parallel or xargs.

Prefer ripgrep (rg) over GNU grep when available; it is typically 3× faster.

Avoid unnecessary sort steps; use awk aggregation when possible.

Never run sed -i directly on production files; create a backup first and validate with nginx -t before reload.

Sanitize any user‑supplied patterns (use -F or proper escaping) to prevent command injection.

Limit resource usage for heavy jobs with ulimit, timeout, or nice.

Real‑Time Monitoring Example

# Simple error tail monitor
tail -f /var/log/app/error.log | grep --line-buffered "ERROR"
# Count recent errors per minute
while true; do
  start=$(date -d '1 minute ago' +%s)
  count=$(awk -v s=$start '$1 >= s && /ERROR/ {c++} END{print c+0}' /var/log/app/app.log)
  if [ "$count" -ge 10 ]; then
    curl -X POST -H "Content-Type: application/json" -d "{\"text\": \"[ALERT] $count errors in last minute\"}" https://your-webhook-url
  fi
  sleep 60
done

Conclusion

The author emphasizes four key takeaways:

Choose the right tool: grep for fast search, sed for in‑place edits, awk for complex aggregation.

Apply performance optimizations—filter early, use LC_ALL=C, parallelize large workloads, and consider ripgrep for speed.

Prioritize safety—always backup before modifying production files and sanitize external inputs.

Debug methodically—inspect a few lines with head, print intermediate fields, and verify delimiters before scaling up.

By mastering these three classic utilities, SREs can handle most log‑analysis tasks without resorting to heavyweight platforms, keeping troubleshooting fast, reliable, and portable.