Master Server Log Analysis with Essential Linux Commands

Running a personal website on Alibaba Cloud ECS, the author shares useful command-line snippets for analyzing Apache access logs.

Basic IP and Page Statistics

<code>awk '{print $1}' log_file|sort|uniq|wc -l</code>

Count the number of unique IP addresses.

<code>grep "/index.php" log_file | wc -l</code>

Count how many times a specific page was accessed.

<code>awk '{++S[$1]} END {for (a in S) print a,S[a]}' log_file > log.txt
sort -n -t ' ' -k 2 log.txt</code>

Show how many pages each IP accessed.

<code>awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n</code>

List IPs sorted by the number of pages they accessed.

<code>grep ^111.111.111.111 log_file| awk '{print $1,$7}'</code>

Show which pages a particular IP visited.

<code>awk '{print $12,$1}' log_file | grep ^"Mozilla | awk '{print $2}' |sort | uniq | wc -l</code>

Exclude search engine crawlers from the count.

<code>awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}'| sort | uniq | wc -l</code>

Count unique IPs within a specific hour.

<code>awk '{print $1}' |sort|uniq -c|sort -nr |head -10</code>

Show the top 10 IP addresses by request count.

uniq -c groups and places the count at the beginning.

<code>cat access.log|awk '{print $1}'|sort|uniq -c|sort -nr|head -10</code>

Another way to list the top 10 IPs.

<code>cat access.log|awk '{counts[$(11)]+=1}; END {for(url in counts) print counts[url], url}</code>

List the most requested files or pages.

<code>cat access.log | awk '{print $11}' | sed -e ' s/http:\/\///' -e ' s/\/.*//' | sort | uniq -c | sort -rn | head -20</code>

Count accesses by subdomain using the referer field.

<code>cat www.access.log |awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}'|sort -nr|head -100</code>

List files with the largest transfer size.

<code>cat www.access.log |awk '($10 > 200000 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100</code>

Find pages larger than 200 KB and how often they were requested.

<code>cat www.access.log |awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}'|sort -nr|head -100</code>

Show pages with the longest response time when the last column records transfer time.

<code>cat www.access.log |awk '($NF > 60 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100</code>

List pages that took more than 60 seconds.

<code>cat www.access.log |awk '($NF > 30){print $7}'|sort -n|uniq -c|sort -nr|head -20</code>

List pages with transfer time over 30 seconds.

<code>ps -ef | awk -F ' ' '{print $8 " " $9}' |sort | uniq -c |sort -nr |head -20</code>

Show the number of processes per command, sorted descending.

<code>netstat -an | grep ESTABLISHED | wc -l</code>

Count current established connections (Apache concurrent requests).

<code>netstat -nat|grep -i "80"|wc -l</code>

Total number of requests on port 80.

<code>netstat -na|grep ESTABLISHED|wc -l</code>

Number of established TCP connections.

<code>netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'</code>

Display per‑IP connection counts and totals for each TCP state.

LAST_ACK 5 – connection closing; SYN_RECV 30 – pending requests; ESTABLISHED 1597 – normal data transfer; FIN_WAIT1 51 – server closing; FIN_WAIT2 504 – client closing; TIME_WAIT 1057 – waiting after close.

These commands provide a comprehensive toolbox for monitoring traffic, performance, and security of a web server.