Master Server Log Analysis: 20 Essential Linux Commands to Uncover Traffic and Issues

Running a personal website on an Alibaba Cloud ECS instance sparked the need to occasionally inspect server logs for traffic patterns and potential malicious activity; the following commands provide a comprehensive toolkit for such analysis.

1. Count distinct IP addresses: awk '{print $1}' log_file | sort | uniq | wc -l 2. Count accesses to a specific page (e.g., /index.php): grep "/index.php" log_file | wc -l 3. Show how many pages each IP accessed:

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt

4. Sort IPs by the number of pages visited (ascending):

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

5. List pages visited by a specific IP (e.g., 111.111.111.111):

grep ^111.111.111.111 log_file | awk '{print $1, $7}'

6. Exclude search‑engine crawlers from page counts:

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

7. Count unique IPs within a specific hour (e.g., 16:00 on 16 Aug 2015):

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Show the top 10 most frequent IPs:

awk '{print $1}' access_log | sort | uniq -c | sort -nr | head -10

9. Identify the 10 most requested files or pages:

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

10. Count accesses per sub‑domain (using the Referer field):

cat access.log | awk '{print $11}' | sed -e 's/http:\/\///' -e 's/\/.*//' | sort | uniq -c | sort -rn | head -20

11. List files with the largest transferred size:

cat www.access.log | awk '($7~/.php/){print $10 " " $1 " " $4 " " $7}' | sort -nr | head -100

12. Find pages larger than 200 KB and their request counts:

cat www.access.log | awk '($10 > 200000 && $7~/.php/){print $7}' | sort | uniq -c | sort -nr | head -100

13. Show the slowest pages when the last column records transfer time:

cat www.access.log | awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}' | sort -nr | head -100

14. List pages taking more than 60 seconds and their frequencies:

cat www.access.log | awk '($NF > 60 && $7~/\.php/){print $7}' | sort | uniq -c | sort -nr | head -100

15. List files with transfer time over 30 seconds:

cat www.access.log | awk '($NF > 30){print $7}' | sort | uniq -c | sort -nr | head -20

16. Show the number of processes per executable, sorted descending:

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

17. Check current Apache concurrent connections: netstat -an | grep ESTABLISHED | wc -l 18. Count Apache processes (each request may spawn a process): ps -ef | grep httpd | wc -l 19. Summarize connections per IP and overall connection states:

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":"); if(n<=2) ++S[array[1]]; else ++S[array[4]]; ++s[$NF]; ++N} END {for(a in S){printf("%-20s %s
", a, S[a]); ++I} printf("%-20s %s
","TOTAL_IP",I); for(a in s) printf("%-20s %s
",a, s[a]); printf("%-20s %s
","TOTAL_LINK",N);}'

20. Additional useful queries (examples):

Top 20 URLs on 04/May/2012:

cat access.log | grep '04/May/2012' | awk '{print $11}' | sort | uniq -c | sort -nr | head -20

IPs that accessed www.abc.com:

cat access_log | awk '($11~/www.abc.com/){print $1}' | sort | uniq -c | sort -nr

Hourly IP connection peaks:

awk -vFS='[:]' '{gsub("-.*","",$1); num[$2 " " $1]++} END {for(i in num) print i,num[i]}' log_file | sort -n -k3 -r | head -10

Calculate total traffic in GB:

cat access.log | awk '{sum+=$10} END {print sum/1024/1024/1024}'

Count 404 responses: awk '($9~/404/){print $9,$7}' access.log | sort Show HTTP status distribution:

cat access.log | awk '{counts[$9]++} END {for(code in counts) print code, counts[code]}'

Per‑second request rate for specific status codes:

watch "awk '{if($9~/200|30|404/)COUNT[$4]++} END{for(a in COUNT) print a,COUNT[a]}' log_file | sort -k2 -nr | head -10"

These commands together form a robust toolbox for administrators to monitor traffic, detect anomalies, and optimize server performance.