How to Build a High‑Performance MyBatis Log Analyzer with Spring Boot

Introduction

Recently I encountered a problem in a project: MyBatis log files in production become huge, making manual slow‑SQL search extremely difficult. To facilitate log analysis I built a log‑analysis system.

Project Background

Problems Encountered

Huge log files: Production logs can be several hundred MB, manual search is like finding a needle in a haystack.

Inconsistent formats: Different projects have slightly different log formats, requiring flexible adaptation.

High performance requirements: Need to process large files quickly without affecting the running system.

Multiple analysis dimensions: Need to locate slow SQL and also analyse execution time, parameters, frequency, etc.

Technology Selection

After research I chose the following stack:

Backend: Spring Boot + Java 8+ (stable, rich ecosystem)

Frontend: Vue.js + Element UI (high development efficiency, rich components)

Database: MySQL (lightweight, easy to deploy)

Core algorithm: Regular expressions + multithreading (excellent performance)

System Architecture Design

Overall Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Frontend UI   │    │   Backend API   │    │   Database      │
│   Vue.js        │◄──►│   Spring Boot   │◄──►│   MySQL          │
│   Element UI    │    │   MyBatis       │    │   log_template   │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Core Modules

Log parsing engine: Parses different log formats.

Template management system: Supports custom log‑format templates.

Performance monitoring system: Real‑time monitoring of processing performance and system status.

Result display system: Provides a friendly UI and data export.

Core Technical Implementation

1. High‑Performance Log Parsing

Multithreaded batch processing

// Batch processing to avoid OOM
List<Future<Void>> futures = new ArrayList<>();
for (int i = 0; i < lines.size(); i += BATCH_SIZE) {
    final int startIndex = i;
    int endIndex = Math.min(i + BATCH_SIZE, lines.size());
    List<String> batch = lines.subList(i, endIndex);
    Future<Void> future = executorService.submit(() -> {
        processBatch(batch, startIndex, sqlExecutionMap, slowSqlResults);
        return null;
    });
    futures.add(future);
}

Intelligent cache management

// Use ConcurrentHashMap for concurrency
Map<String, SqlExecutionInfo> sqlExecutionMap = new ConcurrentHashMap<>();

// Automatic cache cleanup
private void cleanupExpiredCache(Map<String, SqlExecutionInfo> sqlExecutionMap) {
    long currentTime = System.currentTimeMillis();
    long expireTime = currentTime - 300000; // 5 minutes
    sqlExecutionMap.entrySet().removeIf(entry -> {
        SqlExecutionInfo info = entry.getValue();
        return info.getStartTime() != null &&
               info.getStartTime().toEpochSecond(java.time.ZoneOffset.UTC) * 1000 < expireTime;
    });
}

2. Flexible Regular‑Expression Templates

Template design

The system supports three main regex templates:

Preparing template: Matches SQL preparation statements.

Parameters template: Matches SQL parameters.

Total template: Matches SQL execution results.

// MyBatis standard format templates
String preparingRegex = "(?<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) \\[(?<thread>[^\\]]+)\\] DEBUG (?<class>[\\w\\.]+) - ==>  Preparing: (?<sql>.+)";
String parametersRegex = "(?<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) \\[(?<thread>[^\\]]+)\\] DEBUG (?<class>[\\w\\.]+) - ==> Parameters: (?<params>.+)";
String totalRegex = "(?<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) \\[(?<thread>[^\\]]+)\\] TRACE (?<class>[\\w\\.]+) - <==  Total: (?<total>\\d+)";

Multi‑format support

The system includes built‑in templates for three log formats:

MyBatis standard format: Most common MyBatis logs.

Spring Boot format: Supports various time formats.

Nginx access log: Enables HTTP request performance analysis.

3. Performance Monitoring System

Monitoring metrics

@Component
public class LogAnalysisPerformanceMonitor {
    // Performance statistics
    private final AtomicLong totalProcessedLines = new AtomicLong(0);
    private final AtomicLong totalProcessedBytes = new AtomicLong(0);
    private final AtomicLong totalProcessingTime = new AtomicLong(0);
    private final AtomicInteger totalSlowSqlCount = new AtomicInteger(0);
    private final AtomicInteger totalFilesProcessed = new AtomicInteger(0);
    // Regex statistics
    private final AtomicLong totalRegexMatches = new AtomicLong(0);
    private final AtomicLong totalRegexAttempts = new AtomicLong(0);
    // Cache statistics
    private final AtomicLong maxCacheSize = new AtomicLong(0);
    private final AtomicLong totalCacheCleanups = new AtomicLong(0);
}

Real‑time performance report

The system generates detailed reports including file count, byte count, line count, throughput, processing speed, average time, regex match rate, cache usage, and error statistics.

Pitfalls and Solutions

1. Regex matching issues

Problem: Initial regex had low match rate, many SQL statements were not recognized.

Cause: Slight differences in log format, inconsistent time formats, special character escaping.

Solution:

// Support multiple time formats in regex
private String adjustTimeRegex(String regex) {
    // Support with or without milliseconds
    return regex.replace("(?<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2})",
                         "(?<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}(?::\\d{3}|\\.\\d{3})?)");
}

2. Memory overflow

Problem: Processing large files caused OOM.

Cause: Loading entire file into memory, cache not cleaned, thread pool not controlled.

Solution:

// Batch processing large files
private static final int BATCH_SIZE = 1000;
private static final int MAX_CACHE_SIZE = 10000;

// NIO file reading
private List<String> readFileLines(MultipartFile file) throws IOException {
    List<String> lines = new ArrayList<>();
    try (BufferedReader reader = new BufferedReader(
            new InputStreamReader(file.getInputStream(), StandardCharsets.UTF_8))) {
        String line;
        while ((line = reader.readLine()) != null) {
            lines.add(line);
        }
    }
    return lines;
}

Performance Optimization Practices

1. Multithreaded optimization

Thread pool size: Dynamically adjust based on CPU cores.

Batch size: Balance memory usage and processing efficiency.

Task allocation: Reasonable distribution to avoid thread starvation.

2. Memory optimization

Object reuse: Reduce object creation and GC pressure.

Cache strategy: LRU cache with timely eviction.

Streaming processing: Process large files as streams to avoid OOM.

3. Algorithm optimization

Pre‑filtering: Skip irrelevant lines before regex matching.

Index optimization: Use Map structures for fast lookups.

Batch operations: Reduce database round‑trips.

Actual Effect

Performance Metrics

Processing speed: Average 55,000 lines/s.

Throughput: 4 MB/s.

Memory usage: Controlled between 500 MB‑2 GB.

Accuracy: SQL match accuracy > 95 %.

Features

Multi‑format support: MyBatis, Spring Boot, Nginx logs.

Intelligent analysis: Automatic slow‑SQL detection with detailed reports.

Real‑time monitoring: Live performance updates and optimization suggestions.

Data export: CSV export for further analysis.

Project Summary

Business Value

Efficiency boost: Automation improves slow‑SQL locating efficiency by more than tenfold.

Problem定位: Quickly pinpoint performance issues, reducing troubleshooting time.

Performance optimization: Provides data support for DB tuning.

Operations support: Gives ops teams a powerful log analysis tool.

Future Plans

Machine learning: Introduce ML to automatically identify abnormal SQL patterns.

Real‑time analysis: Support streaming log analysis.

Visualization enhancement: Add more charts and statistics.

Distributed support: Enable cluster deployment and distributed processing.

Source Code

https://gitee.com/jack_fly/analysis_log