20 Powerful Shell Commands to Master Log File Analysis

The article shares a collection of twenty practical shell commands for analyzing web server log files, enabling administrators to uncover traffic patterns, detect potential attacks, and monitor system performance.

1. Count unique IP addresses

awk '{print $1}' log_file | sort | uniq | wc -l

2. Count accesses to a specific page

grep "/index.php" log_file | wc -l

3. Count pages visited per IP

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt
sort -n -t ' ' -k2 log.txt

4. Sort IPs by number of pages visited

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

5. List pages accessed by a specific IP

grep ^111.111.111.111 log_file | awk '{print $1, $7}'

6. Exclude search‑engine crawlers

awk '{print $12, $1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

7. Count IPs that accessed during a specific hour

awk '{print $4, $1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Top 10 IPs by request count

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

9. Top 10 most requested files or pages

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

10. Top 20 URLs by sub‑domain (using referer)

cat access.log | awk '{print $11}' | sed -e 's/http:\/\///' -e 's/\/.*//' | sort | uniq -c | sort -nr | head -20

11. Largest transferred files

cat www.access.log | awk '($10 > 200000 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

12. Pages larger than 200 KB and their frequencies

cat www.access.log | awk '($10 > 200000 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

13. Slowest PHP pages (duration > 60 s)

cat www.access.log | awk '($NF > 60 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

14. Pages with response time > 30 s

cat www.access.log | awk '($NF > 30){print $7}' | sort -n | uniq -c | sort -nr | head -20

15. Process count per service

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

16. Current Apache concurrent connections

netstat -an | grep ESTABLISHED | wc -l

17. Compare with MaxClients setting

netstat -an | grep ESTABLISHED | wc -l

18. Count Apache processes handling requests

ps -ef | grep httpd | wc -l

19. Total 80‑port connections

netstat -nat | grep -i "80" | wc -l

20. Detailed TCP state statistics

netstat -n | awk '/^tcp/ {++S[$NF]} END {for (a in S) print a, S[a]}'

These commands can be combined, filtered, and piped to generate custom reports such as the most active IPs, busiest URLs on a given day, bandwidth usage, HTTP status distribution, and connection state summaries, providing a solid foundation for log‑driven troubleshooting and security monitoring.