Master Server Log Analysis: 20 Essential Linux Commands to Uncover Traffic, Errors, and Performance

1. Count distinct IP addresses awk '{print $1}' log_file | sort | uniq | wc -l 2. Count visits to a specific page grep "/index.php" log_file | wc -l 3. Show how many pages each IP accessed

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt
sort -n -t ' ' -k2 log.txt

4. Sort IPs by the number of pages they accessed (ascending)

awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n

5. List pages visited by a specific IP

grep ^111.111.111.111 log_file | awk '{print $1,$7}'

6. Exclude search‑engine crawlers from statistics

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq -c | wc -l

7. Count IPs that accessed the site during a specific hour (e.g., 14:00 on 16 Aug 2015)

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq -c | wc -l

8. Show the top 10 IP addresses by request count

awk '{print $1}' access_log | sort | uniq -c | sort -nr | head -10

9. List the 10 most requested files or pages

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

10. Count accesses per sub‑domain using the Referer field

cat access.log | awk '{print $11}' | sed -e 's/http:////' -e 's//.*//' | sort | uniq -c | sort -rn | head -20

11. List files with the largest transfer size

cat www.access.log | awk '($7~/.php/){print $10,$1,$4,$7}' | sort -nr | head -100

12. Show pages larger than 200 KB and their hit counts

cat www.access.log | awk '($10>200000 && $7~/.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

13. Identify the slowest PHP pages (by response time)

cat www.access.log | awk '($7~/.php/){print $NF,$1,$4,$7}' | sort -nr | head -100

14. List PHP pages taking more than 60 seconds and their frequencies

cat www.access.log | awk '($NF>60 && $7~/.php/){print $7}' | sort | uniq -c | sort -nr | head -100

15. Find files whose transfer time exceeds 30 seconds

cat www.access.log | awk '($NF>30){print $7}' | sort | uniq -c | sort -nr | head -20

16. Show the number of processes per command (sorted descending)

ps -ef | awk -F ' ' '{print $8,$9}' | sort | uniq -c | sort -nr | head -20

17. Get the current Apache concurrent connection count netstat -an | grep ESTABLISHED | wc -l 18. Compare the actual concurrent connections with MaxClients setting ps -ef | grep httpd | wc -l 19. Summarize connections per IP and overall TCP state counts

netstat -n | awk '/^tcp/ {n=split($(NF-1),a,":"); if(n<=2) ++S[a[1]]; else ++S[a[4]]; ++state[$NF]; ++total} END {for (i in S) printf "%‑20s %s", i, S[i]; printf "%‑20s %s", "TOTAL_IP", total; for (s in state) printf "%‑20s %s", s, state[s]; printf "%‑20s %s", "TOTAL_LINK", total}'

20. Additional useful queries (e.g., top URLs on a specific date, IPs requesting a given domain, per‑minute traffic spikes, etc.)

# Top 20 URLs on 04/May/2012
cat access.log | grep '04/May/2012' | awk '{print $11}' | sort | uniq -c | sort -nr | head -20

# IPs that accessed www.abc.com
cat access_log | awk '($11~/www\.abc\.com/){print $1}' | sort | uniq -c | sort -nr

# Hourly request distribution
awk '{print $1}' access.log | grep '20/Mar/2011' | cut -c14-18 | sort | uniq -c | sort -nr | head

These commands together form a practical toolbox for administrators and developers who need to monitor web traffic, detect anomalies, diagnose performance issues, and secure their services by spotting suspicious access patterns.