Skip to content
Master Server Log Analysis with Powerful Bash One‑Liners
Operations 13 min read

Master Server Log Analysis with Powerful Bash One‑Liners

This guide presents a comprehensive set of Bash and AWK one‑liners for analyzing web server logs, counting unique IPs, tracking page visits, sorting traffic by time, identifying heavy‑weight requests, monitoring Apache processes, and examining network connection states, helping you detect anomalies and optimize performance.

MaGe Linux Operations
MaGe Linux Operations
MaGe Linux Operations

Running a personal website on Alibaba Cloud ECS, I occasionally analyze server logs to monitor traffic and detect malicious activity. Below is a collection of useful shell commands for various log‑analysis tasks.

1. Count how many unique IPs accessed the site: awk '{print $1}' log_file | sort | uniq | wc -l 2. Count how many times a specific page was accessed (e.g., /index.php): grep "/index.php" log_file | wc -l 3. Show how many pages each IP visited:

awk '{++S[$1]} END {for (a in S) print a, S[a]}' log_file > log.txt
sort -n -t ' ' -k2 log.txt

4. Sort IPs by the number of pages they visited (ascending):

awk '{++S[$1]} END {for (a in S) print S[a], a}' log_file | sort -n

5. List pages visited by a specific IP (replace 111.111.111.111):

grep ^111.111.111.111 log_file | awk '{print $1,$7}'

6. Exclude requests from search engine crawlers:

awk '{print $12,$1}' log_file | grep ^"Mozilla" | awk '{print $2}' | sort | uniq | wc -l

7. Count IP accesses within a specific hour (e.g., 14:00 on 16 Aug 2015):

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}' | sort | uniq | wc -l

8. Show the top 10 IP addresses by request count:

awk '{print $1}' log_file | sort | uniq -c | sort -nr | head -10

9. List the 10 most requested files or pages:

awk '{print $11}' log_file | sort | uniq -c | sort -nr | head -10

10. Count accesses per sub‑domain using the Referer field (approximate):

awk '{print $11}' access.log | sed -e 's/http:\/\///' -e 's/\/.*//' | sort | uniq -c | sort -rn | head -20

11. List files with the largest transferred size:

awk '($7~/.php/){print $10, $1, $4, $7}' www.access.log | sort -nr | head -100

12. Show pages larger than 200 KB and their request counts:

awk '($10 > 200000 && $7~/.php/){print $7}' www.access.log | sort | uniq -c | sort -nr | head -100

13. Find the pages that took the longest to deliver to the client (assuming the last column is response time):

awk '($7~/.php/){print $NF, $1, $4, $7}' www.access.log | sort -nr | head -100

14. List pages whose response time exceeds 60 seconds and their frequencies:

awk '($NF > 60 && $7~/.php/){print $7}' www.access.log | sort | uniq -c | sort -nr | head -100

15. List files with response time over 30 seconds:

awk '($NF > 30){print $7}' www.access.log | sort | uniq -c | sort -nr | head -20

16. Show the number of running processes per command, sorted descending:

ps -ef | awk -F ' ' '{print $8 " " $9}' | sort | uniq -c | sort -nr | head -20

17. Check current Apache concurrent connections (compare with MaxClients): netstat -an | grep ESTABLISHED | wc -l 18. Count Apache processes (each request may spawn a process): ps -ef | grep httpd | wc -l 19. Show connection counts per IP and total connections per state:

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":"); if(n<=2) ++S[array[1]]; else ++S[array[4]]; ++s[$NF]; ++N} END {for(a in S){printf("%-20s %s", a, S[a]); ++I} printf("%-20s %s","TOTAL_IP",I); for(a in s) printf("%-20s %s",a, s[a]); printf("%-20s %s","TOTAL_LINK",N);}'

20. Additional useful snippets (e.g., traffic by hour, bandwidth, HTTP status codes, time‑wait connections, SYN counts, etc.) are included throughout the original article.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Apachebashlog analysisNetwork Monitoringserver administrationawk
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.