Master Server Log Analysis with Essential Linux Commands

I run a personal website on an Alibaba Cloud ECS server and occasionally analyze its access logs to monitor traffic and spot potential attacks. Below is a curated list of useful Linux commands for log analysis.

1. Count unique visitor IPs: awk '{print $1}' log_file | sort | uniq | wc -l 2. Count visits to a specific page (e.g., /index.php): grep "/index.php" log_file | wc -l 3. Show how many pages each IP accessed:

awk '{++S[$1]} END {for (a in S) print a,S[a]}' log_file > log.txt
sort -n -t ' ' -k 2 log.txt

4. Sort IPs by the number of pages they accessed (ascending):

awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n

5. List pages visited by a specific IP (replace with the target IP):

grep ^111.111.111.111 log_file | awk '{print $1,$7}'

6. Exclude search‑engine crawlers from the count:

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

7. Count unique IPs within a specific hour (e.g., 14:00 on 16 Aug 2015):

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Show the top 10 IP addresses by request count:

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

Note: uniq -c prefixes each line with its occurrence count.

9. List the 10 most requested files or pages:

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

10. Count requests per sub‑domain (using the Referer header):

cat access.log | awk '{print $11}' | sed -e 's/http:\/\///' -e 's/\/.*//' | sort | uniq -c | sort -rn | head -20

11. Find the largest transferred files:

cat www.access.log | awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}' | sort -nr | head -100

12. List pages larger than 200 KB and their request counts:

cat www.access.log | awk '($10 > 200000 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

13. Identify the slowest pages (by transfer time) if the last column records duration:

cat www.access.log | awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}' | sort -nr | head -100

14. Show pages taking more than 60 seconds and their frequencies:

cat www.access.log | awk '($NF > 60 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

15. Show pages taking more than 30 seconds:

cat www.access.log | awk '($NF > 30){print $7}' | sort -n | uniq -c | sort -nr | head -20

16. List the number of processes per command (sorted descending):

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

17. Check current Apache concurrent connections: netstat -an | grep ESTABLISHED | wc -l 18. Count Apache processes (each request may spawn a process): ps -ef | grep httpd | wc -l 19. Show total connections per IP and overall connection states:

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[1]];else++S[array[4]];++s[$NF];++N} END {for(a in S){printf("%-20s %s
", a, S[a]);++I}printf("%-20s %s
","TOTAL_IP",I);for(a in s) printf("%-20s %s
",a, s[a]);printf("%-20s %s
","TOTAL_LINK",N);}'

20. Find the top 20 URLs on a specific date (e.g., 04 May 2012):

cat access.log | grep '04/May/2012' | awk '{print $11}' | sort | uniq -c | sort -nr | head -20

21. List IPs that accessed a particular domain (e.g., www.abc.com):

cat access_log | awk '($11~/www\.abc\.com/){print $1}' | sort | uniq -c | sort -nr

22. Show the top 10 IPs for a given day and optionally filter by time range:

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{print $1}' | sort | uniq -c | sort -nr | head -10

23. Calculate total traffic in gigabytes:

cat access.log | awk '{sum+=$10} END {print sum/1024/1024/1024}'

24. List all 404 responses:

awk '($9 ~/404/)' access.log | awk '{print $9,$7}' | sort

25. Summarize HTTP status codes:

cat access.log | awk '{counts[$9]++} END {for(code in counts) print code, counts[code]}'
cat access.log | awk '{print $9}' | sort | uniq -c | sort -rn

26. Show per‑second request rates for specific status codes:

watch "awk '{if($9~/200|30|404/)COUNT[$4]++} END {for(a in COUNT) print a,COUNT[a]}' log_file | sort -k 2 -nr | head -n10"

27. Estimate client request count (simple bandwidth metric):

cat apache.log | awk '{if($7~/GET/) count++} END {print "client_request="count}'

28. Identify the 10 IPs with the most accesses on a particular day:

cat /tmp/access.log | grep "20/Mar/2011" | awk '{print $3}' | sort | uniq -c | sort -nr | head

29. Find what the most active IP was doing on a given day:

cat access.log | grep "10.0.21.17" | awk '{print $8}' | sort | uniq -c | sort -nr | head -n10

30. Determine the 10 busiest hour‑long intervals (by IP connections):

awk -vFS=":" '{gsub("-.*","",$1); num[$2" "$1]++} END {for(i in num) print i,num[i]}' log_file | sort -n -k 3 -r | head -10

31. Find the minutes with the highest request volume:

awk '{print $1}' access.log | grep "20/Mar/2011" | cut -c 14-18 | sort | uniq -c | sort -nr | head

32. Monitor TCP connection states in real time:

watch "netstat -n | awk '/^tcp/ {++S[\$NF]} END {for(a in S) print a, S[a]}'"

Common TCP states: LAST_ACK (connection closing), SYN_RECV (waiting for processing), ESTABLISHED (normal data transfer), FIN_WAIT1 (server initiates close), FIN_WAIT2 (client aborts), TIME_WAIT (awaiting timeout).

Source: segmentfault.com