Master Server Log Analysis: 30 Essential Linux Commands for Monitoring Traffic

Running a personal website on an Alibaba Cloud ECS instance? Periodically analyzing server logs can reveal traffic patterns, potential attacks, and performance bottlenecks. Below is a curated collection of Linux commands for extracting valuable insights from Apache (or similar) access logs.

Basic Traffic Statistics

1. Count unique visitor IPs: awk '{print $1}' log_file | sort | uniq | wc -l 2. Count accesses to a specific page (e.g., index.php): grep "/index.php" log_file | wc -l 3. Show how many pages each IP accessed:

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt

4. Sort IPs by number of pages visited (ascending):

awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n

Detailed Per‑IP Queries

5. List pages visited by a particular IP (replace with actual IP):

grep ^111.111.111.111 log_file | awk '{print $1,$7}'

6. Exclude search‑engine crawlers and count distinct pages:

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

Time‑Based Queries

7. IP count for a specific hour (e.g., 16/Aug/2015:14):

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Top 10 IPs by request count:

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

Popular Resources

9. Top 10 most requested files/pages:

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

10. Requests per subdomain (based on referer):

cat access.log | awk '{print $11}' | sed -e 's|http://||' -e 's|/.*||' | sort | uniq -c | sort -rn | head -20

Bandwidth and Size Analysis

11. Largest transferred files (by size):

cat www.access.log | awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}' | sort -nr | head -100

12. Pages larger than 200 KB and their hit counts:

cat www.access.log | awk '($10 > 200000 && $7~/\.php/){print $7}' | sort | uniq -c | sort -nr | head -100

Performance Timing

13. Pages with longest client‑side transfer time (if last column records time):

cat www.access.log | awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}' | sort -nr | head -100

14. Pages taking more than 60 seconds:

cat www.access.log | awk '($NF > 60 && $7~/\.php/){print $7}' | sort | uniq -c | sort -nr | head -100

15. Pages exceeding 30 seconds:

cat www.access.log | awk '($NF > 30){print $7}' | sort | uniq -c | sort -nr | head -20

Process and Connection Monitoring

16. Count of each running process (sorted):

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

17. Current Apache concurrent connections: netstat -an | grep ESTABLISHED | wc -l 18. Total Apache processes (example output shows 1388): ps -ef | grep httpd | wc -l 19. Connection count per IP and overall state totals:

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":"); if(n<=2) ++S[array[1]]; else ++S[array[4]]; ++s[$NF]; ++N} END {for(a in S){printf("%-20s %s
", a, S[a]); ++I} printf("%-20s %s
","TOTAL_IP",I); for(a in s) printf("%-20s %s
",a, s[a]); printf("%-20s %s
","TOTAL_LINK",N);}'

HTTP Status and Error Tracking

20. Count of each HTTP status code:

cat access.log | awk '{counts[$9]++} END {for(code in counts) print code, counts[code]}'

21. List of 404 errors with requested URLs:

awk '($9 ~/404/)' access.log | awk '{print $9,$7}' | sort

Advanced Time‑Window Queries

22. Log entries for specific dates (e.g., 15/Aug/2015 and 16/Aug/2015):

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{print $1}' | sort | uniq -c | sort -nr | head -10

23. IPs that accessed a particular URL during that window:

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{if($7 == "/index.php?g=Member&m=Public&a=sendValidCode") print $1,$7}' | sort | uniq -c | sort -nr

These snippets provide a quick toolbox for routine log inspection, security auditing, and performance tuning on Linux‑based web servers.