Master Server Log Analysis: 20 Essential Linux Commands to Uncover Traffic Insights

Below is a collection of useful Linux commands for analyzing your web server logs (Apache access logs). They help you monitor traffic, detect anomalies, and gather performance metrics.

1. Count unique IP addresses

awk '{print $1}' log_file | sort | uniq | wc -l

2. Count accesses to a specific page

grep "/index.php" log_file | wc -l

3. Count pages visited per IP

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt
sort -n -t ' ' -k2

4. Sort IPs by number of pages visited (ascending)

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

5. List pages visited by a specific IP

grep ^111.111.111.111 log_file | awk '{print $1, $7}'

6. Exclude search‑engine crawlers

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

7. Count IPs that accessed during a specific hour (e.g., 14:00 on 16 Aug 2015)

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Top 10 IP addresses by request count

awk '{print $1}' log_file | sort -nr | uniq -c | sort -nr | head -10

9. Top 10 most requested URLs

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

10. Top 20 IPs by request count (using cat )

cat log_file | awk '{print $1}' | sort | uniq -c | sort -nr | head -20

11. List files with largest transfer size (PHP files)

cat www.access.log | awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}' | sort -nr | head -100

12. List pages larger than 200 KB and their request counts

cat www.access.log | awk '($10 > 200000 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

13. Pages with longest response time (last field is time)

cat www.access.log | awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}' | sort -nr | head -100

14. Pages taking more than 60 seconds

cat www.access.log | awk '($NF > 60 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

15. Files with transfer time > 30 seconds

cat www.access.log | awk '($NF > 30){print $7}' | sort -n | uniq -c | sort -nr | head -20

16. Top 20 processes by instance count

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

17. Current Apache concurrent connections

netstat -an | grep ESTABLISHED | wc -l

18. Compare with MaxClients setting

netstat -an | grep -i "80" | wc -l

19. Detailed TCP connection statistics

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":"); if(n<=2) ++S[array[1]]; else ++S[array[4]]; ++s[$NF]; ++N} END {for(a in S) printf("%-20s %s
", a, S[a]); printf("%-20s %s
","TOTAL_IP",N); for(a in s) printf("%-20s %s
",a,s[a]); printf("%-20s %s
","TOTAL_LINK",N);}'

20. Miscellaneous useful queries

Top 20 URLs on 04/May/2012:

cat access.log | grep '04/May/2012' | awk '{print $11}' | sort | uniq -c | sort -nr | head -20

IPs that accessed URLs containing www.abc.com:

cat access_log | awk '($11~/www\.abc\.com/){print $1}' | sort | uniq -c | sort -nr

Requests to /index.php?g=Member&m=Public&a=sendValidCode on 15‑16 Aug 2015:

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{if($7=="/index.php?g=Member&m=Public&a=sendValidCode") print $1,$7}' | sort | uniq -c | sort -nr

Count 404 responses: awk '($9~/404/){print $9,$7}' access.log | sort HTTP status distribution:

cat access.log | awk '{counts[$9]++} END {for(code in counts) print code, counts[code]}'

Per‑second concurrency (example):

watch "awk '{if($9~/200|30|404/) COUNT[$4]++} END {for(a in COUNT) print a,COUNT[a]}' log_file | sort -k2 -nr | head -n10"

Bandwidth usage (GB):

cat access.log | awk '{sum+=$10} END {print sum/1024/1024/1024}'

These commands provide a solid toolbox for daily log inspection, performance tuning, and security monitoring of Apache or other web servers.