Stop Using cat Blindly: Practical Linux Commands for GB‑Scale Backend Log Troubleshooting

Background

Production logs in modern distributed Java services can reach dozens of gigabytes, contain interleaved multi‑thread output, long stack traces, and are rotated frequently. Quickly locating the root cause of timeouts or crashes requires efficient use of native Linux log tools.

Production‑grade log characteristics and pitfalls

Log rotation – Logs are often renamed (e.g., app.log → app.log.1.gz) when they exceed a size or time threshold. Analysts must handle compressed files instead of relying on basic view commands.

Performance and safety of view commands – Opening huge files with vi or vim loads the entire file into memory, causing I/O blockage or OOM. Choose among cat, less and tail based on depth and real‑time needs.

Distributed tracing – In micro‑service environments a single TraceId injected via MDC links logs across services. The first step in troubleshooting is to fetch the TraceId and then search all relevant log files.

Cross‑file aggregation : grep -h "trace-123456" *.log | sort extracts matching records from business, middleware and audit logs and sorts them by timestamp to reconstruct the request chain.

Trace completeness check :

find /var/log/myapp -name "*.log" | xargs grep "trace-123456"

quickly finds missing links across the log directory.

Core practical commands

grep – the workhorse (≈99% usage)

grep (Global Regular Expression Print)

searches text with regular expressions. Commonly used options: -A n / -B n / -C n – print n lines after, before, or both sides of a match (essential for capturing full Java exception stacks). -v – invert match (e.g., grep -v "healthCheck" to filter heartbeat lines). -i – ignore case (matches both exception and Exception). -E – enable extended regex (e.g., grep -E "[1-9]+"). -h – hide filenames when aggregating multiple files.

Example – list all lines containing a specific TraceId and sort by time: grep -h '2037009686800355328' *.log | sort If timestamps are not at the line start, sort by the timestamp field:

grep '2036988356092674048' *.log | sort -t '[' -k4,4

Capture surrounding context:

grep -A50 '2036706972224598017' app-error.log

tail – real‑time monitoring

tail

shows the last N lines of a file. In production, tail -F is preferred because it automatically reopens the file after rotation.

-c, --bytes=NUM   output last NUM bytes
-f, --follow[={name|descript}]   follow file growth
-F                same as --follow=name --retry
-n, --line=NUM    output last NUM lines
--pid=<pid>        exit when the given PID ends
-q, --quiet       suppress filenames when multiple files
--retry           keep trying to open a file that becomes unavailable
-s, --sleep-interval=<seconds>   interval between checks
-v, --verbose     always show filenames

Typical usage:

tail -n 100 -f app.log

sed – stream editing without loading the whole file

sed

processes input line‑by‑line, making it safe for GB‑scale logs. Extract a time window:

sed -n '/2026-03-26 10:15/,/2026-03-26 10:20/p' app-info.log

Tip: match to minute precision to avoid endless output when the exact second has no entries.

awk – column‑wise analysis

awk

treats each line as a record split by spaces or tabs, enabling structured analysis of access logs, Tomcat logs, etc. awk '$NF > 1.0 {print $1, $7, $NF}' access.log Key built‑ins: $0 – entire line $1…$n – individual fields $NF – last field (useful when column count varies) NR – current record number FS – field separator (changeable with -F)

less – lazy loading for massive files

less

reads data only when the user scrolls, providing fast navigation of multi‑GB logs. Interactive shortcuts: /pattern – forward search ?pattern – backward search n / N – next/previous match Space / b – page down/up G / g – jump to end/start &pattern – show only matching lines -N – display line numbers

Real‑time follow mode: less +F filename Press Ctrl+C to return to normal browsing, then F to resume follow.

Viewing compressed logs

Production logs are often gzipped. Linux provides transparent tools that operate on compressed streams: zgrep – search compressed archives (e.g., find a TraceId across multi‑day .gz files). zless – view a week‑old archive without explicit decompression. zcat – pipe a compressed log to awk or sed for offline analysis. zmore – quick flip‑through of compressed files.

Practical pipelines (combination techniques)

Combining basic commands yields powerful one‑liners for common scenarios.

Combo 1 – Count specific business executions in a time window

sed -n '/15:00/,/15:30/p' app-info.log | grep "逾期任务自动撤销" | wc -l

Combo 2 – Top‑10 most frequent exceptions

grep "Exception" app-error.log | awk '{print $NF}' | sort | uniq -c | sort -nr | head -n 10

Combo 3 – Capture slow requests (>1 s) and list their paths

awk '$NF > 1.0 {print $7}' access.log | sort | uniq -c | sort -nr | head -n 10

Combo 4 – Needle‑in‑haystack across directories

find /var/log/myapp -name "*.log" | xargs grep "2036706972224598017"

Summary

Even when centralized log platforms (ELK, SkyWalking) are available, native Linux tools remain indispensable for situations where those services are delayed, crashed, or when direct access to isolated environments is required. Mastering grep, tail, sed, awk, less and their compressed‑file counterparts provides a low‑overhead, precise “Swiss‑army‑knife” for production log analysis.