Master Server Log Analysis with Powerful Bash One‑Liners

Running a personal website on Alibaba Cloud ECS, I occasionally analyze server logs to monitor traffic and detect malicious activity. Below is a collection of useful shell commands for various log‑analysis tasks.

1. Count how many unique IPs accessed the site: awk '{print $1}' log_file | sort | uniq | wc -l 2. Count how many times a specific page was accessed (e.g., /index.php): grep "/index.php" log_file | wc -l 3. Show how many pages each IP visited:

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt
sort -n -t ' ' -k2 log.txt

4. Sort IPs by the number of pages they visited (ascending):

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

5. List pages visited by a specific IP (replace 111.111.111.111):

grep ^111.111.111.111 log_file | awk '{print $1,$7}'

6. Exclude requests from search engine crawlers:

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

7. Count IP accesses within a specific hour (e.g., 14:00 on 16 Aug 2015):

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Show the top 10 IP addresses by request count:

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

9. List the 10 most requested files or pages:

awk '{print $11}' log_file | sort | uniq -c | sort -nr | head -10

10. Count accesses per sub‑domain using the Referer field (approximate):

awk '{print $11}' access.log | sed -e 's/http:\/\///' -e 's/\/.*//' | sort | uniq -c | sort -rn | head -20

11. List files with the largest transferred size:

awk '($7~/.php/){print $10, $1, $4, $7}' www.access.log | sort -nr | head -100

12. Show pages larger than 200 KB and their request counts:

awk '($10 > 200000 && $7~/.php/){print $7}' www.access.log | sort | uniq -c | sort -nr | head -100

13. Find the pages that took the longest to deliver to the client (assuming the last column is response time):

awk '($7~/.php/){print $NF, $1, $4, $7}' www.access.log | sort -nr | head -100

14. List pages whose response time exceeds 60 seconds and their frequencies:

awk '($NF > 60 && $7~/.php/){print $7}' www.access.log | sort | uniq -c | sort -nr | head -100

15. List files with response time over 30 seconds:

awk '($NF > 30){print $7}' www.access.log | sort | uniq -c | sort -nr | head -20

16. Show the number of running processes per command, sorted descending:

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

17. Check current Apache concurrent connections (compare with MaxClients): netstat -an | grep ESTABLISHED | wc -l 18. Count Apache processes (each request may spawn a process): ps -ef | grep httpd | wc -l 19. Show connection counts per IP and total connections per state:

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":"); if(n<=2) ++S[array[1]]; else ++S[array[4]]; ++s[$NF]; ++N} END {for(a in S){printf("%-20s %s", a, S[a]); ++I} printf("%-20s %s","TOTAL_IP",I); for(a in s) printf("%-20s %s",a, s[a]); printf("%-20s %s","TOTAL_LINK",N);}'

20. Additional useful snippets (e.g., traffic by hour, bandwidth, HTTP status codes, time‑wait connections, SYN counts, etc.) are included throughout the original article.