Master Log File Searching: Powerful Grep & Awk Commands for Massive Logs
This guide shows how to efficiently search and count matching lines in huge log files using grep, awk, and regular expressions, covering basic commands, combined conditions, port filtering, and IP address extraction for faster log analysis.
Introduction
When dealing with very large log files, opening them with editors like vim or using cat can freeze the terminal, yet you often need to know how many lines match a certain pattern. Below are useful command‑line techniques for fast log searching.
Common Search Commands
grep search
grep 参数 文件名 | head // from the beginninggrep 参数 文件名 | wc -l // count matching linescat 文件名 | grep 参数$ // lines ending with the parameterExamples
Count lines containing a specific request:
cat /data/weblogs/xxx.access.log | grep "GET /pixel.jpg?" | wc -lResult: 4102386 Partial regex query for a time range:
cat /data/weblogs/em.evony.com.access.log | grep "25/Nov/2019:15:[00-59]" | wc -lResult: 120 Combine multiple conditions with pipes:
cat /data/weblogs/xxx.log | grep "25/Nov/2019:15:[00-59]" | grep "GET /pixel.jpg?" | wc -lResult: 120 OR condition using grep -E:
cat /data/weblogs/xxx.log | grep -E "25/Nov/2019:15:[00-59] |GET /pixel.jpg?" | wc -lResult:
4098135Fuzzy Grep for Ports
Simple grep on netstat may return many unrelated lines: netstat -anp | grep -i '80' A more precise command filters only port 80 processes:
netstat -apn | awk '{split($4,arr,":"); if(arr[2]=="80") print $0}'This prints only the processes using port 80.
Searching for IP Addresses
Match IPs
grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' xxx.log | wc -lResult:
31116275Count occurrences of each IP
grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" xxx.log | sort | uniq -cSample output:
2 99.203.87.1032 99.203.87.1424 99.203.87.1458 99.203.87.153More precise IP matching
grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" xxx.log | wc -lResult:
32929372Fuzzy IP matching
grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" xxx.log | wc -lResult:
32930309Multiple conditions then IP count
cat xxx.log | grep "25/Nov/2019:15:[00-59]" | grep "GET /pixel.jpg?" | grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" | wc -lResult: 1110 These methods help you quickly extract line counts, filter by ports, and analyze IP occurrences even when log files continuously grow.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
