Stop Decompressing Logs: Search Compressed Files Directly with zgrep and rg -z

Why Avoid Decompression Loops

Production logs are often split by day or size, resulting in dozens of .gz files per day. The naive approach — writing a for loop to decompress each file, grep, then delete — wastes disk I/O and time.

Core Tools for Direct Compressed Search

zgrep + wildcard — simplest, built-in, reads .gz directly and supports wildcards.

zcat + xargs — suitable for complex pipelines where you need to pipe decompressed content to other commands.

ripgrep (rg) with -z — fastest, multi-threaded, reads .gz, .bz2, .xz, .zstd without separate commands.

Scenario: Find All "TimeoutException" Across 7 Days of Logs

Log files follow the pattern /data/logs/app/app.log.2026-05-25.gz through 2026-06-01.gz.

1. Simplest: zgrep with wildcard

zgrep "TimeoutException" /data/logs/app/app.log.2026-05-*.gz

Outputs filename and matching lines. Use zgrep -h to suppress filenames.

2. Count occurrences per file

zgrep -c "TimeoutException" /data/logs/app/app.log.2026-05-*.gz

Example output:

app.log.2026-05-25.gz:23
app.log.2026-05-26.gz:45
app.log.2026-05-27.gz:12
...

Immediately shows which day had the most timeouts.

3. Mixed .gz and .log files

zgrep "TimeoutException" /data/logs/app/app.log*

zgrep

treats non-compressed files as plain text; no special handling needed.

Advanced: Pipelines with zcat

When you need more than grep — e.g., awk extraction, sort, uniq — pipe zcat output:

zcat /data/logs/app/app.log.2026-05-*.gz | awk '/TimeoutException/ {print $5}' | sort | uniq -c | sort -nr | head -10

zcat

concatenates multiple files automatically; no need for gunzip -c per file.

Speed Boost: ripgrep -z

rg

does not search compressed files by default; add -z:

rg -z "TimeoutException" /data/logs/app/app.log.2026-05-*.gz

Options: --no-filename to hide filenames, -c for counts. Supports .gz, .bz2, .xz, .zstd uniformly.

Cross-Day Time-Range Extraction

Requirement: search 2026-05-25 14:00 to 2026-05-26 10:00.

zcat /data/logs/app/app.log.2026-05-{25,26}.gz | awk '$0 >= "2026-05-25 14:00:00" && $0 <= "2026-05-26 10:00:00" && /keyword/'

Assumes timestamp at line start with consistent format. For many days, use brace expansion:

files=/data/logs/app/app.log.2026-05-{25,26,27,28,29,30,31}.gz
zcat $files | awk ...

Reusable Script: log-scan-days

Place in /usr/local/bin/log-scan-days:

#!/bin/bash
# Usage: log-scan-days /path/to/logs PREFIX START_DATE END_DATE PATTERN
# Example: log-scan-days /data/logs/app app.log 2026-05-25 2026-05-31 "TimeoutException"

LOG_DIR=$1
PREFIX=$2
START=$3
END=$4
PATTERN=$5

d=$START
files=""
while [[ "$d" < "$END" || "$d" == "$END" ]]; do
  file="${LOG_DIR}/${PREFIX}.${d}.gz"
  if [[ -f "$file" ]]; then
    files="$files $file"
  fi
  d=$(date -I -d "$d + 1 day")
done

if [[ -z "$files" ]]; then
  echo "No matching log files found"
  exit 1
fi

if command -v rg &> /dev/null; then
  rg -z "$PATTERN" $files
else
  zgrep "$PATTERN" $files
fi

Run:

log-scan-days /data/logs/app app.log 2026-05-25 2026-05-31 "TimeoutException"

Script: Daily Count CSV (log-error-per-day)

#!/bin/bash
# Usage: log-error-per-day /path/to/logs PREFIX START END PATTERN

LOG_DIR=$1
PREFIX=$2
START=$3
END=$4
PATTERN=$5

d=$START
while [[ "$d" < "$END" || "$d" == "$END" ]]; do
  file="${LOG_DIR}/${PREFIX}.${d}.gz"
  if [[ -f "$file" ]]; then
    count=$(zgrep -c "$PATTERN" "$file")
    echo "$d,$count"
  fi
  d=$(date -I -d "$d + 1 day")
done

Output:

2026-05-25,23
2026-05-26,45
2026-05-27,12
...

Import directly into Excel for trend charts.

Performance Comparison (10 × 1 GB .gz files)

zgrep

wildcard: ~12 seconds zcat | grep: ~11 seconds rg -z: ~4 seconds

Conclusion: rg -z is fastest, but zgrep is perfectly adequate if rg is unavailable.

Common Pitfalls

Argument list too long with thousands of files. Solution:

find /data/logs/app -name "app.log.*.gz" | xargs zgrep "ERROR"

zcat loses filenames — it merges streams. Use zgrep -H or rg -z -H to retain file context.

Timestamp boundaries across files — zcat concatenates, so awk time filtering works globally. For per-file stats, process files individually (as in the daily-count script).

Quick Reference

Search keyword in all compressed files: zgrep "ERROR" /path/*.gz Count matches per file: zgrep -c "ERROR" /path/*.gz Pipe multiple files to awk: zcat /path/*.gz | awk '/ERROR/ {print $1}' Fastest search (install rg): rg -z "ERROR" /path/*.gz Too many files error: find /path -name "*.gz" -exec zgrep "ERROR" {} + Date-range scan (script): log-scan-days (see above)

Remember: zgrep *.gz and rg -z replace manual decompression loops entirely.