Master Server Log Analysis with Powerful Bash Commands

Analyzing web server logs is essential for understanding traffic patterns, detecting attacks, and optimizing performance. The following Bash snippets demonstrate how to extract useful metrics from typical Apache/Nginx access logs.

Basic IP and Page Statistics

Count distinct visitor IPs: awk '{print $1}' log_file | sort | uniq | wc -l Count accesses to a specific page (e.g., /index.php): grep "/index.php" log_file | wc -l Show how many pages each IP requested:

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file | sort -k2 -n

Sort IPs by the number of pages they accessed (ascending):

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

Filtering and Excluding Unwanted Requests

Remove entries generated by search‑engine bots (e.g., Mozilla):

awk '{print $12,$1}' log_file | grep "^\"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

Time‑Based Queries

IP count for a specific hour (e.g., 14:00 on 16 Aug 2015):

awk '{print $4,$1}' log_file | grep "16/Aug/2015:14" | awk '{print $2}' | sort | uniq | wc -l

Top 10 IPs in a given period:

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{print $1}' | sort | uniq -c | sort -nr | head -10

Top Pages and Files

Most requested files (top 10):

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

Largest transferred files (by byte size, PHP only):

cat www.access.log | awk '($7~/\.php/){print $10, $1, $4, $7}' | sort -nr | head -100

Pages whose transfer size exceeds 200 KB (PHP only):

cat www.access.log | awk '($10>200000 && $7~/\.php/){print $7}' | sort | uniq -c | sort -nr | head -100

Response Time and Status Codes

Requests taking more than 60 seconds (PHP pages):

cat access.log | awk '($NF>60 && $7~/\.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

Count of each HTTP status code:

cat access.log | awk '{counts[$9]++} END {for (code in counts) print code, counts[code]}'

Bandwidth and Traffic Volume

Total transferred data in gigabytes:

cat access.log | awk '{sum+=$10} END {print sum/1024/1024/1024}'

Connection and Process Monitoring (netstat/ps)

Current ESTABLISHED TCP connections: netstat -an | grep ESTABLISHED | wc -l Count of connections per TCP state:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for (a in S) print a, S[a]}'

Top 20 IPs with most connections on port 80:

netstat -ant | awk '/:80/ {split($5,ip,":"); ++A[ip[1]]} END {for (i in A) print A[i], i}' | sort -rn | head -20

Processes using the most CPU (example for Apache):

ps -ef | awk -F' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

These one‑liners can be combined, piped, or wrapped in scripts to build automated log‑analysis pipelines, helping administrators quickly spot anomalies, heavy hitters, and performance bottlenecks.