Essential Linux Commands for Analyzing Web Server Logs

Count unique visitor IPs

Extract the first field (IP) from the log, sort, deduplicate and count the lines:

awk '{print $1}' log_file | sort | uniq | wc -l

Count accesses to a specific page

Search for the page path and count matching lines:

grep "/index.php" log_file | wc -l

How many pages each IP visited

Aggregate by IP and output the number of requests per IP:

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt
sort -n -t ' ' -k2 log.txt

Sort IPs by request count (ascending)

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

List pages requested by a particular IP

grep ^111.111.111.111 log_file | awk '{print $1, $7}'

Exclude search‑engine crawlers

Keep only entries whose user‑agent starts with "Mozilla" and count unique URLs:

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq -c | wc -l

Count IPs during a specific hour

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

Top 10 IPs by request volume

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

Find the 10 most requested URLs

cat log_file | awk '{print $11}' | sort | uniq -c | sort -nr | head -10

Identify the largest transferred files (by size field $10)

cat www.access.log | awk '($7 ~ /.php/){print $10, $1, $4, $7}' | sort -nr | head -100

Pages whose transfer time exceeds 60 seconds

cat www.access.log | awk '($NF > 60 && $7 ~ /.php/){print $7}' | sort -n | uniq -c | sort -nr | head -100

Bandwidth usage (GB)

cat access.log | awk '{sum+=$10} END {print sum/1024/1024/1024}'

Count HTTP status codes

cat access.log | awk '{counts[$9]++} END {for (code in counts) print code, counts[code]}'

Current TCP connection states

Show the number of sockets in each state:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for (a in S) print a, S[a]}'

Top IPs with ESTABLISHED connections on port 80

netstat -an | grep ESTABLISHED | wc -l

Find IPs with many TIME_WAIT sockets

netstat -n | grep TIME_WAIT | awk '{print $5}' | sort | uniq -c | sort -rn | head -20

Detect potential attackers (most frequent source IPs on port 80)

netstat -anlp | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20

These snippets can be combined, filtered, or extended to suit specific forensic or performance‑monitoring needs on any Linux‑based web server.