Essential Linux Commands for Analyzing Web Server Logs

Below is a curated set of one‑liners and pipelines that can be run on typical Apache/Nginx access logs (e.g., access.log or log_file) to answer common operational questions.

Basic IP and request statistics

Count distinct visitor IPs: awk '{print $1}' log_file | sort | uniq | wc -l Show how many pages each IP accessed: awk '{++S[$1]} END {for (ip in S) print ip, S[ip]}' log_file List top 10 IPs by request count:

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

Page‑level analysis

Requests for a specific page (e.g., /index.php): grep "/index.php" log_file | wc -l Top 10 most requested URLs:

awk '{print $11}' log_file | sort | uniq -c | sort -nr | head -10

Pages sorted by total bytes transferred (assuming the size field is $10): awk '{print $10,$7}' log_file | sort -nr | head -20 Pages whose response time exceeds 60 s (assuming the time field is $NF):

awk '($NF>60 && $7~/\.php/){print $7}' log_file | sort -n | uniq -c | sort -nr | head -20

Time‑range queries

Requests in a specific hour (e.g., 14:00 on 16 Aug 2015):

awk '{print $4,$1}' log_file | grep "16/Aug/2015:14" | awk '{print $2}' | sort | uniq | wc -l

IP list for a given two‑day window:

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{print $1}' | sort | uniq -c | sort -nr | head -10

HTTP status and bandwidth

Count of each HTTP status code:

awk '{counts[$9]++} END {for (code in counts) print code, counts[code]}' log_file

Total traffic in gigabytes (field $10 holds bytes): awk '{sum+=$10} END {print sum/1024/1024/1024}' log_file Number of 404 responses:

awk '($9~/404/){print}' log_file | wc -l

TCP connection state monitoring

Current number of ESTABLISHED connections: netstat -an | grep ESTABLISHED | wc -l Breakdown of TCP states:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for (s in S) print s, S[s]}'

Top source IPs on port 80:

netstat -ant | awk '/:80/{split($5,ip,":"); ++cnt[ip[1]]} END {for (ip in cnt) print cnt[ip], ip}' | sort -nr | head -20

Count of TIME_WAIT sockets:

netstat -n | grep TIME_WAIT | awk '{print $5}' | sort | uniq -c | sort -rn | head -20

Additional useful one‑liners

Show per‑process request count for Apache: ps -ef | grep httpd | wc -l List processes with most open TCP connections:

ps -ef | awk -F' ' '{print $8,$9}' | sort | uniq -c | sort -nr | head -20

Live monitoring of TCP state changes:

watch "netstat -n | awk '/^tcp/ {++S[\$NF]} END {for(s in S) print s, S[s]}'"

These snippets can be combined, filtered, or extended with sed, awk, and sort to suit specific investigative needs, providing a quick, script‑free way to audit web traffic, detect anomalies, and gauge server health.