grep, awk, sort: Analyze 10GB Logs in 5 Minutes (Scripts Included)

Why Not Use vim for Huge Logs

Opening multi-gigabyte log files with vim or less saturates disk I/O and freezes the terminal. The author advocates streaming command-line filters that never load the entire file into memory.

Core Toolset: grep/awk/sort

grep/zgrep

: keyword search, compressed log search awk: field splitting, time-range filtering, data calculation sort/uniq/wc: sorting, deduplication, counting

Chaining these with pipes outperforms GUI log viewers.

Five High-Frequency Scenarios

1. Count Log Levels

grep -oE '(INFO|WARN|ERROR)' app.log | sort | uniq -c | sort -nr

Sample output:

54321 INFO
1234 WARN
567 ERROR

Low ERROR count suggests errors are not the primary cause of slowdown.

2. Top 20 Most Frequent Errors

grep "ERROR" app.log | sed 's/.*] //' | sort | uniq -c | sort -nr | head -20

sed 's/.*] //'

strips the timestamp prefix inside brackets. For logs like 2026-05-31 14:23:45,123 ERROR com.xxx - message, use cut -d' ' -f5- or awk '{for(i=5;i<=NF;i++) printf "%s ", $i}'. To skip indented stack traces, prepend grep -v '^[[:space:]]'; if rg is available, use --multiline for cross-line matching.

3. Top 10 Client IPs from Gateway Logs

grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort | uniq -c | sort -nr | head -10

Immediately identifies high-frequency callers for bot or abuse investigation.

4. Time-Range Extraction (Pitfall Warning)

The common awk '/start/,/end/' pattern is unreliable: if the end timestamp is missing, it prints to EOF, and regexes can match unrelated lines. The author prefers string comparison, which requires timestamps at line start in lexicographically sortable format (e.g., yyyy-MM-dd HH:mm:ss):

awk '$0 >= "2026-06-10 14:00:00" && $0 <= "2026-06-10 14:30:00"' app.log

Two hard requirements: timestamp at line start, uniform format. Minute precision allows shorthand 2026-06-10 14:00. If timestamps are not at line start, preprocess with grep/sed to extract the time field first.

5. Faster Search: rg or Optimized grep

ripgrep (rg)

uses multi-threading and skips binary files, delivering 2-5x speedup over grep.

# Install
apt install ripgrep   # Ubuntu
yum install ripgrep   # CentOS

# Usage
rg "ERROR" app.log --no-line-number -c          # count error lines
rg "NullPointer" app.log -A 5 -B 5              # show 5 lines before/after

When third-party tools are forbidden, set LC_ALL=C to bypass Unicode parsing, nearly doubling grep throughput:

LC_ALL=C grep "ERROR" app.log

Reusable Scripts

Place in /usr/local/bin and chmod +x for global access.

log-stat — Log Level Counts

#!/bin/bash
# Usage: log-stat app.log
grep -oE '(INFO|WARN|ERROR|DEBUG)' "$1" | sort | uniq -c | sort -nr

log-top-error — Top N Errors

#!/bin/bash
# Usage: log-top-error app.log [lines, default 20]
lines=${2:-20}
grep "ERROR" "$1" | sed 's/.*] //' | sort | uniq -c | sort -nr | head -$lines

log-time — Time-Range Extract

#!/bin/bash
# Usage: log-time app.log "2026-06-10 14:00:00" "2026-06-10 14:30:00"
# Precondition: timestamp at line start, uniform format
start=$2
end=$3
awk -v s="$start" -v e="$end" '$0 >= s && $0 <= e' "$1"

Search Compressed Logs Without Decompression

Logrotate archives logs as .gz or .zst. Use zgrep / zcat for gzip, zstdgrep for zstd — same syntax as grep.

# gzip
zcat app.log.1.gz | head -100
zgrep "ERROR" app.log.1.gz
zgrep -c "ERROR" app.log.1.gz

# zstd
zstdgrep "ERROR" app.log.1.zst

All previous grep pipelines work by swapping grep for zgrep.

Monitor Progress with pv

For 10GB files, pv shows real-time throughput, processed volume, and ETA:

pv app.log | grep "ERROR" | wc -l

Real Incident: 6GB Export Timeout Resolved in 5 Minutes

Count errors: rg "ERROR" export.log -c → 234 errors, not systemic.

Categorize:

rg "ERROR" export.log | sed 's/.*] //' | sort | uniq -c | sort -nr | head -5

→ DB connection pool exhausted 189 times.

Trace SQL: rg -B 5 'exhausted' export.log | grep 'SQL' | sort | uniq -c → one full-table-scan query executed 180+ times.

Fix: increase pool size, add index on the table.

Evidence handed directly to developers, eliminating blame-shifting.

Windows Local Debugging

Avoid slow native findstr /PowerShell. Two reliable options:

Install Git for Windows — includes bash with grep/awk/sort.

Install WSL with Ubuntu — full Linux environment locally.

Quick Reference Cheat Sheet

Count log levels :

grep -oE '(INFO|WARN|ERROR)' app.log | sort | uniq -c | sort -nr

Top errors :

grep ERROR app.log | sed 's/.*] //' | sort | uniq -c | sort -nr | head -20

Top IPs :

grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort | uniq -c | sort -nr | head -10

Extract time range :

awk '$0 >= "2026-05-31 14:00:00" && $0 <= "2026-05-31 14:30:00"' app.log

Search .gz : zgrep ERROR app.log.1.gz Search .zst : zstdgrep ERROR app.log.1.zst Ultra-fast search : Install ripgrep use rg; no permission → LC_ALL=C grep Show progress :

pv app.log | grep ERROR | wc -l