Master Log File Searching: Powerful Grep & Awk Commands for Massive Logs

Introduction

When dealing with very large log files, opening them with editors like vim or using cat can freeze the terminal, yet you often need to know how many lines match a certain pattern. Below are useful command‑line techniques for fast log searching.

Common Search Commands

grep search

grep 参数 文件名 | head          // from the beginning

grep 参数 文件名 | wc -l          // count matching lines

cat 文件名 | grep 参数$          // lines ending with the parameter

Examples

Count lines containing a specific request:

cat /data/weblogs/xxx.access.log | grep "GET /pixel.jpg?" | wc -l

Result: 4102386 Partial regex query for a time range:

cat /data/weblogs/em.evony.com.access.log | grep "25/Nov/2019:15:[00-59]" | wc -l

Result: 120 Combine multiple conditions with pipes:

cat /data/weblogs/xxx.log | grep "25/Nov/2019:15:[00-59]" | grep "GET /pixel.jpg?" | wc -l

Result: 120 OR condition using grep -E:

cat /data/weblogs/xxx.log | grep -E "25/Nov/2019:15:[00-59] |GET /pixel.jpg?" | wc -l

Result:

4098135

Fuzzy Grep for Ports

Simple grep on netstat may return many unrelated lines: netstat -anp | grep -i '80' A more precise command filters only port 80 processes:

netstat -apn | awk '{split($4,arr,":"); if(arr[2]=="80") print $0}'

This prints only the processes using port 80.

Searching for IP Addresses

Match IPs

grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' xxx.log | wc -l

Result:

31116275

Count occurrences of each IP

grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" xxx.log | sort | uniq -c

Sample output:

2 99.203.87.103

2 99.203.87.142

4 99.203.87.145

8 99.203.87.153

More precise IP matching

grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" xxx.log | wc -l

Result:

32929372

Fuzzy IP matching

grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" xxx.log | wc -l

Result:

32930309

Multiple conditions then IP count

cat xxx.log | grep "25/Nov/2019:15:[00-59]" | grep "GET /pixel.jpg?" | grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" | wc -l

Result: 1110 These methods help you quickly extract line counts, filter by ports, and analyze IP occurrences even when log files continuously grow.