Master Linux Log Analysis: tail, less, grep, sed, awk Combo for Production Debugging

tail

Many beginners use cat to view logs, but for large files cat floods the terminal and can freeze it. tail is the proper tool for real-time monitoring.

Scenario A: Service Deployment Monitoring

During each deployment restart, we need to confirm Spring Boot started successfully or catch initialization errors.

# -f (follow): continuously display appended lines at the end of the file
tail -f logs/application.log

Scenario B: Reproducing Bugs with QA

QA says: "I'm clicking the button now, check if the backend throws an error." At this moment we only need to watch the latest output, not historical logs.

# Show only the last 200 lines and keep refreshing in real time, avoiding interference from historical logs
tail -n 200 -f logs/application.log

less

When you need to examine earlier logs, less is recommended. Unlike vim which loads the entire file into memory, less loads on demand, making it extremely smooth even for multi-gigabyte files, and it supports backward navigation.

Scenario: Investigating a Customer Complaint Order

Operations reports: "Around 10:00, order ORD12345678 failed payment." You need to search backward from the end of the log for this order ID. less logs/application.log Operations after entering less: Shift + G – jump to the very end of the log (errors usually occur recently). ?ORD12345678 – type question mark + order ID to search upward (reverse) . n – if the current match isn't the key information, press n to continue searching upward for the previous occurrence. Shift + F – if new logs arrive while browsing, this key combo switches less into a real-time scrolling mode similar to tail -f; press Ctrl + C to return to browse mode.

grep

grep

is the most common search command, but simple keyword searches are often insufficient in real business scenarios.

Scenario A: Restoring Error Context (Key)

Seeing only a NullPointerException line rarely pinpoints the problem; we need to know the request parameters before the error and the stack trace after it. This requires the -C (Context) parameter.

# Search for the exception keyword and show 20 lines before and after each match
grep -C 20 "NullPointerException" logs/application.log

Scenario B: Full-Chain TraceId Search

Microservices typically use TraceId to correlate requests. Log files may have rolled (e.g., app.log, app.log.1, app.log.2). We need to search the same TraceId across all log files.

# Search all files starting with app.log in the current directory
grep "TraceId-20251219001" logs/app.log*

Scenario C: Counting Exception Frequency

Management asks: "How many Redis timeout exceptions occurred today? Is it sporadic or widespread?" No manual counting needed; just count matching lines.

# -c (count): only output the number of matching lines
grep -c "RedisConnectionException" logs/application.log

Scenario D: Excluding Noise

During troubleshooting, logs are cluttered with irrelevant INFO heartbeat or health-check lines that severely distract.

# -v (invert): show all lines that do NOT contain "HealthCheck"
grep -v "HealthCheck" logs/application.log

sed

When logs are huge (e.g., 10 GB), grep output may still be overwhelming. If we know the production incident occurred between 14:00 and 14:05 , sed can extract just that time window into a small file for offline analysis.

Scenario: Exporting Logs for a Specific Time Window

# Syntax: sed -n '/start_time/,/end_time/p' source_file > target_file
# Note: the time format must exactly match the format in the log lines
sed -n '/2025-12-19 14:00/,/2025-12-19 14:05/p' logs/application.log > error_segment.log

This yields a compact error_segment.log (only a few MB) that can be downloaded locally or shared with colleagues for further analysis.

awk

awk

excels at processing columnar data. For well-structured logs like Nginx access logs or Apache logs, it can generate summary reports directly on the server.

Scenario A: Identifying Malicious IPs During an Attack

Service suddenly alerts on CPU spike, suspected CC attack or scraper. We analyze Nginx logs to find the top offending IPs. Assuming the first column is the client IP:

# 1. awk '{print $1}': extract the first column (IP)
# 2. sort: sort lines so identical IPs are adjacent
# 3. uniq -c: deduplicate and count occurrences per IP
# 4. sort -nr: sort by count (n) in reverse (r) order
# 5. head -n 10: take the top 10

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -n 10

Scenario B: Finding Slowest Endpoints

Nginx logs often record response time in the last column. We want to list requests where response time exceeds 1 second (assuming URL is in column 7).

# $NF represents the last column
# Print URL and response time for all entries where response time > 1.000 seconds
awk '$NF > 1.000 {print $7, $NF}' access.log

Summary

The examples above are my daily go-to commands. I recommend memorizing them or bookmarking this article so that next time a production issue arises, you can match the scenario and copy-paste the command directly.