How to Uncover Hidden Threats: Mastering Web Log Analysis with Secsoso

Why Analyze Web Logs

As web technologies evolve, websites become critical business services, making their behavior and security paramount. Analyzing web logs helps identify security events, support incident forensics, and understand common business actions.

How to Perform Log Analysis

Web servers generate logs for each request. For example, an Apache access log entry looks like:

<code>114.221.137.87 - - [10/Sep/2019:14:52:28 +0800] "GET /login HTTP/1.1" 200 1068 https://secilog.secisland.com/ Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.105 Safari/537.36 Vivaldi/2.4.1488.40</code>

From such entries we can differentiate normal user requests from malicious scans or attacks by looking for patterns such as numerous 404 responses for non‑existent URLs or suspicious query strings.

Log Analysis Scenarios

Scenario 1: Identify the time window of an intrusion, then examine suspicious logs within that range to pinpoint the attacker and reconstruct the attack flow.

Scenario 2: After an intrusion, attackers often leave backdoor files; locating these files provides clues for deeper analysis.

While experienced engineers can use shell commands, most users benefit from a dedicated tool. The example below uses the free tool Secsoso to demonstrate common queries.

Business Behavior Statistics

Top 10 IPs by Access Count

<code>secsoso 'file=("/export/home/20190613/access.log"," ")|stats count($1) by $1|sort 10 -count_$1'</code>

Result shows IP addresses and their request counts.

Count of Distinct IPs

<code>secsoso 'access.log|rename $1 as ip |stats dc(ip)'</code>

Page‑Specific Access Count

<code>secsoso 'access.log " /nodejs-underscore/?cf_action=sync_comments"|eventcount'</code>

Pages Visited per IP

<code>secsoso 'access.log $7="*.html*" |stats $1 as ip,count($1) as count by $1'</code>

Hourly Access Volume

<code>secsoso 'access.log|eval date=$4.to_date("[dd/MMM/yyyy:HH:mm:ss")|stats count(date) by tspan(date,"1h")'</code>

Browser Fingerprint Filtering

<code>secsoso 'access.log "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9) Gecko Minefield/3.0" $7!="*.js*" $7!="*.css*"|fields $1,$4,$7'</code>

Crawler Statistics

<code>secsoso 'access.log "Googlebot|Baiduspider|Applebot|bingbot|YoudaoBot|Sosospider|MSNBot"|stats dc($1) as count'</code>

IP Segment Access Count

<code>secsoso 'access.log|stats rex($1,"[0-3]+\.[0-9]+\.[0-9]+") as ipper,count(1) as count by ipper|sort -count'</code>

Status Code Distribution

<code>secsoso 'access.log|stats $9 as stat ,count($9) as count by $9'</code>

URL Access Frequency

<code>secsoso 'access.log|eval url=$7.substring(0,POSITION("?",$7))|stats count(url) by url'</code>

Security Statistics

CC Attack Detection (excluding static resources)

<code>secsoso 'access.log $7!="/.*\.js.*|.*\.css.*|.*\.png.*/" |eval date=$4.to_date("[dd/MMM/yyyy:HH:mm:ss")|stats $1 as ip,count(date) as count by tspan(date,"1h"),$1|search count>50'</code>

CC Attack Detection (including static resources)

<code>secsoso 'access.log|eval date=$4.to_date("[dd/MMM/yyyy:HH:mm:ss")|stats $1 as ip,count(date) as count by tspan(date,"1h"),$1|search count>100'</code>

SQL Injection Attempts

<code>secsoso 'access.log "%20select%20|%20and%201=1|%20and%201=2|%20exec|%27exec|information_schema.tables|%20where%20|%20union%20|%20SELECT%20|%2ctable_name%20|cmdshell|%20table_schema"|fields $1,$7'</code>

SQL Injection IP Ranking

<code>secsoso 'access.log "%20select%20|%20and%201=1|%20and%201=2|%20exec|%27exec|information_schema.tables|%20where%20|%20union%20|%20SELECT%20|%2ctable_name%20|cmdshell|%20table_schema"|fields $1,$7|stats $1 as ip,count(1) as count by $1|sort -count'</code>

Sensitive File Scanning

<code>secsoso 'access.log ".zip|.rar|.mdb|.inc|.sql|.config|.bak|/login.inc.php|.svn|/mysql/|config.inc.php|.bak|wwwroot|网站备份|/gf_admin/|/DataBackup/|/Web.config|/web.config|/1.txt|/test.txt"|fields $1,$7'</code>

Vulnerability Exploitation

<code>secsoso 'access.log "struts|jmx-console|ajax_membergroup.php|iis.txt|phpMyAdmin|getWriter|dirContext|phpmyadmin|acunetix.txt|/e/|/SouthidcEditor/|/DatePicker/"|fields $1,$7'</code>

File Inclusion Attacks

<code>secsoso 'access.log "/passwd|%00|/win.ini|/my.ini|/MetaBase.xml|/ServUDaemon.ini|cmd.exe"|fields $1,$7'</code>

Getshell Attacks

<code>secsoso 'access.log " eval|%eval|%execute|%3binsert|%20makewebtaski|/1.asp|/1.jsp|/1.php|/1.aspx|/xiaoma.jsp|/tom.jsp|/py.jsp|/k8cmd.jsp|/k8cmd|/ver007.jsp|/ver008.jsp|/ver007|/ver008|\.aar|%if"|fields $1,$7'</code>

XSS (Cross‑Site Scripting) Attacks

<code>secsoso 'access.log "(S)%3C(S+)%3E|(S)%3C(S+)%2F%3E|(S+)&lt;(S+)&gt;|(S+)&lt;(S+)/&gt;|onerror|onmouse|expression|alert|document.|prompt\()"|fields $1,$7'</code>

Login Password Guessing

Detect IPs with more than ten failed login attempts within five minutes:

<code>secsoso 'access.log $6="*POST" $7="*/login" $9="200"|eval date=$4.to_date("[dd/MMM/yyyy:HH:mm:ss")|stats count($1) as count by tspan(date,"5m"),$1|search count>10|stats sum(count) by $1'</code>

The analysis demonstrates that web logs contain rich information for security monitoring, but effective use requires preserving request parameters (including POST data) and applying systematic queries.