Transforming Log Anomaly Detection with Text Clustering

1 Introduction

Log analysis is a primary fault‑diagnosis method throughout the software lifecycle. Traditional bank systems rely on manual inspection and regex‑based automated reporting, which become ineffective as log volume grows.

When logs reach tens of gigabytes per hour, manual search is infeasible.

Regex‑based automation requires developers to write extensive patterns, which are hard to maintain.

Frequent application updates render existing regexes obsolete.

Regexes often generate false alarms, e.g., normal messages like “Error check completed.” being mis‑identified.

Frequency‑based anomalies cannot be captured without prior knowledge of the log patterns.

To address these pain points, we propose a text‑clustering approach for intelligent log anomaly identification.

2 Solution Approach for Intelligent Log Detection

2.1 Scenario Analysis

When an incident occurs, engineers typically:

Locate logs around the abnormal time window.

Compare them with normal logs to spot differences.

Check whether the frequency of nearby logs deviates from normal behavior.

The key challenges are:

Deriving normal and abnormal feature sets automatically.

Statistically measuring log frequencies to detect anomalies.

2.2 Proposed Solution

Industry methods such as Drain and SPELL cluster log messages into templates, recording their occurrence frequencies. Our solution builds on this by:

Clustering normal‑time logs and abnormal‑time logs separately, then subtracting the normal feature set from the abnormal set to obtain pure anomaly templates.

Counting template occurrences within a configurable time window to form frequency vectors; deviations from normal vectors trigger alerts.

Implementation steps are as follows.

3 Practical Implementation of Intelligent Log Detection

3.1 Error‑Message Log Detection

Workflow:

Collect normal‑time and abnormal‑time logs as training data.

Apply Drain or SPELL to cluster logs into templates.

Compare abnormal templates with normal ones, retain only the differences as anomaly templates, and store them after verification.

Match incoming logs against the anomaly templates; a match indicates an anomaly.

In a test on 100 k lines of compilation logs, 40 templates were generated.

After clustering, four templates (Event1, Event3, Event4, Event6) appear in the abnormal window; removing those also present in the normal window leaves Event3 and Event6 as suspected anomalies, which are then added to the anomaly model.

New logs are matched against these anomaly templates to flag issues.

3.2 Frequency‑Anomaly Log Detection

Some failures manifest as a sudden surge of a specific log type (e.g., frequent Full GC). The detection steps are:

Cluster normal‑time logs to obtain templates.

Within a defined time window, count occurrences of each template to build a frequency vector.

Generate vectors for all windows, forming a normal feature vector set.

Example vectors:

Normal window vector:

[1,2,0]

During an abnormal GC spike, the vector may become

[10,0,0]

, which deviates significantly from the normal cluster center.

Unsupervised clustering methods such as K‑means, DBSCAN, or Isolation Forest can identify these outlier vectors.

4 Advantages

The model automatically learns abnormal patterns from training data, improving detection accuracy without manual rule definition.

Based on text clustering, it eliminates the need for handcrafted regexes, reducing developer workload and accelerating fault isolation.

Continuous model retraining keeps detection effective across application version updates.