Master Server Log Analysis: 30 Essential Linux Commands for Monitoring Traffic

Running a personal website on an Alibaba Cloud ECS instance? Periodically analyzing server logs can reveal traffic patterns, potential attacks, and performance bottlenecks. Below is a curated collection of Linux commands for extracting valuable insights from Apache (or similar) access logs.

Basic Traffic Statistics

1. Count unique visitor IPs:

<code>awk '{print $1}' log_file | sort | uniq | wc -l</code>

2. Count accesses to a specific page (e.g., index.php):

<code>grep "/index.php" log_file | wc -l</code>

3. Show how many pages each IP accessed:

<code>awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt</code>

4. Sort IPs by number of pages visited (ascending):

<code>awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n</code>

Detailed Per‑IP Queries

5. List pages visited by a particular IP (replace with actual IP):

<code>grep ^111.111.111.111 log_file | awk '{print $1,$7}'</code>

6. Exclude search‑engine crawlers and count distinct pages:

<code>awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l</code>

Time‑Based Queries

7. IP count for a specific hour (e.g., 16/Aug/2015:14):

<code>awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l</code>

8. Top 10 IPs by request count:

<code>awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10</code>

Popular Resources

9. Top 10 most requested files/pages:

<code>cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10</code>

10. Requests per subdomain (based on referer):

<code>cat access.log | awk '{print $11}' | sed -e 's|http://||' -e 's|/.*||' | sort | uniq -c | sort -rn | head -20</code>

Bandwidth and Size Analysis

11. Largest transferred files (by size):

<code>cat www.access.log | awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}' | sort -nr | head -100</code>

12. Pages larger than 200 KB and their hit counts:

<code>cat www.access.log | awk '($10 > 200000 && $7~/\.php/){print $7}' | sort | uniq -c | sort -nr | head -100</code>

Performance Timing

13. Pages with longest client‑side transfer time (if last column records time):

<code>cat www.access.log | awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}' | sort -nr | head -100</code>

14. Pages taking more than 60 seconds:

<code>cat www.access.log | awk '($NF > 60 && $7~/\.php/){print $7}' | sort | uniq -c | sort -nr | head -100</code>

15. Pages exceeding 30 seconds:

<code>cat www.access.log | awk '($NF > 30){print $7}' | sort | uniq -c | sort -nr | head -20</code>

Process and Connection Monitoring

16. Count of each running process (sorted):

<code>ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20</code>

17. Current Apache concurrent connections:

<code>netstat -an | grep ESTABLISHED | wc -l</code>

18. Total Apache processes (example output shows 1388):

<code>ps -ef | grep httpd | wc -l</code>

19. Connection count per IP and overall state totals:

<code>netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":"); if(n<=2) ++S[array[1]]; else ++S[array[4]]; ++s[$NF]; ++N} END {for(a in S){printf("%-20s %s\n", a, S[a]); ++I} printf("%-20s %s\n","TOTAL_IP",I); for(a in s) printf("%-20s %s\n",a, s[a]); printf("%-20s %s\n","TOTAL_LINK",N);}'</code>

HTTP Status and Error Tracking

20. Count of each HTTP status code:

<code>cat access.log | awk '{counts[$9]++} END {for(code in counts) print code, counts[code]}'</code>

21. List of 404 errors with requested URLs:

<code>awk '($9 ~/404/)' access.log | awk '{print $9,$7}' | sort</code>

Advanced Time‑Window Queries

22. Log entries for specific dates (e.g., 15/Aug/2015 and 16/Aug/2015):

<code>cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{print $1}' | sort | uniq -c | sort -nr | head -10</code>

23. IPs that accessed a particular URL during that window:

<code>cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{if($7 == "/index.php?g=Member&m=Public&a=sendValidCode") print $1,$7}' | sort | uniq -c | sort -nr</code>

These snippets provide a quick toolbox for routine log inspection, security auditing, and performance tuning on Linux‑based web servers.