Master Server Log Analysis with Essential Linux Commands

Running a personal website on Alibaba Cloud ECS, the author shares useful command-line snippets for analyzing Apache access logs.

Basic IP and Page Statistics

awk '{print $1}' log_file|sort|uniq|wc -l

Count the number of unique IP addresses. grep "/index.php" log_file | wc -l Count how many times a specific page was accessed.

awk '{++S[$1]} END {for (a in S) print a,S[a]}' log_file > log.txt
sort -n -t ' ' -k 2 log.txt

Show how many pages each IP accessed.

awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n

List IPs sorted by the number of pages they accessed.

grep ^111.111.111.111 log_file| awk '{print $1,$7}'

Show which pages a particular IP visited.

awk '{print $12,$1}' log_file | grep ^"Mozilla | awk '{print $2}' |sort | uniq | wc -l

Exclude search engine crawlers from the count.

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}'| sort | uniq | wc -l

Count unique IPs within a specific hour.

awk '{print $1}' |sort|uniq -c|sort -nr |head -10

Show the top 10 IP addresses by request count.

uniq -c groups and places the count at the beginning.

cat access.log|awk '{print $1}'|sort|uniq -c|sort -nr|head -10

Another way to list the top 10 IPs.

cat access.log|awk '{counts[$(11)]+=1}; END {for(url in counts) print counts[url], url}

List the most requested files or pages.

cat access.log | awk '{print $11}' | sed -e ' s/http:\/\///' -e ' s/\/.*//' | sort | uniq -c | sort -rn | head -20

Count accesses by subdomain using the referer field.

cat www.access.log |awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}'|sort -nr|head -100

List files with the largest transfer size.

cat www.access.log |awk '($10 > 200000 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100

Find pages larger than 200 KB and how often they were requested.

cat www.access.log |awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}'|sort -nr|head -100

Show pages with the longest response time when the last column records transfer time.

cat www.access.log |awk '($NF > 60 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100

List pages that took more than 60 seconds.

cat www.access.log |awk '($NF > 30){print $7}'|sort -n|uniq -c|sort -nr|head -20

List pages with transfer time over 30 seconds.

ps -ef | awk -F ' ' '{print $8 " " $9}' |sort | uniq -c |sort -nr |head -20

Show the number of processes per command, sorted descending. netstat -an | grep ESTABLISHED | wc -l Count current established connections (Apache concurrent requests). netstat -nat|grep -i "80"|wc -l Total number of requests on port 80. netstat -na|grep ESTABLISHED|wc -l Number of established TCP connections.

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s
", a, S[a]);++I}printf("%-20s %s
","TOTAL_IP",I);for(a in s) printf("%-20s %s
",a, s[a]);printf("%-20s %s
","TOTAL_LINK",N);}'

Display per‑IP connection counts and totals for each TCP state.

LAST_ACK 5 – connection closing; SYN_RECV 30 – pending requests; ESTABLISHED 1597 – normal data transfer; FIN_WAIT1 51 – server closing; FIN_WAIT2 504 – client closing; TIME_WAIT 1057 – waiting after close.

These commands provide a comprehensive toolbox for monitoring traffic, performance, and security of a web server.