Cut Alert Troubleshooting Time by 80% with LLM Agents: A Full Technical Walkthrough

Introduction

Typical alert handling requires switching between log platforms, APM dashboards, and tracing systems, consuming 10–30 minutes mainly due to repetitive logins and data stitching. To address this, the team built Troubleshooter , an LLM Agent that automates data gathering, root‑cause analysis, and remediation suggestion generation.

Architecture Design

The system follows a layered design that decouples alert ingestion from investigation. The ingestion layer only receives and persists alerts, while an independent scheduler triggers the investigation workflow.

Technology stack includes Spring AI Alibaba (providing a ready‑made ReAct loop), Spring Boot, and a set of custom tool interfaces.

Core Process

The end‑to‑end investigation chain consists of the following steps:

Alert Ingestion: Receive alert data and generate a unique event ID.

Fingerprint Generation: Extract five dimensions (service name, alert type, error pattern, metric name, error summary) and create a 32‑bit MD5 fingerprint.

Knowledge Matching: Look up similar records in a knowledge base; if a match exists, reuse the historical conclusion.

AI Investigation: A SupervisorAgent orchestrates the ReAct reasoning loop.

Conclusion Validation: A dedicated ValidationAgent checks root‑cause clarity and recommendation completeness.

Report Push: Generate a Markdown report and send it to a Feishu group.

Knowledge Consolidation: Persist confirmed conclusions back to the knowledge base.

AI Investigation Engine: ReAct Agent in Practice

The core uses Spring AI Alibaba's ReactAgent, which implements the classic ReAct cycle:

LLM thinking → select tool → execute tool → observe result → think again → … → output conclusion

Contrary to a simple prompt‑only approach, the SupervisorAgent defines four @Tool methods that the ReAct framework can invoke:

@Component
public class SupervisorAgent {
    // Query logs
    @Tool(description = "Query and analyze application logs, return investigation summary")
    public String queryLogs(String serviceName, Integer minutes, ...) { ... }

    // Query metrics
    @Tool(description = "Query service metrics (QPS/RT/errorRate/CPU/memory/GC)")
    public String queryMetrics(String serviceName, String metricTypes, ...) { ... }

    // Query trace
    @Tool(description = "Query distributed trace by traceId")
    public String queryTrace(String traceId) { ... }

    // Query endpoint errors when no traceId
    @Tool(description = "Query error logs for an endpoint and extract traceIds")
    public String queryEndpointErrors(String serviceName, String endpoint, ...) { ... }

    // Core investigation method
    public InvestigationResult investigate(TroubleEvent event) {
        String dynamicInstruction = buildDynamicInstruction(event);
        ReactAgent agent = ReactAgent.builder()
            .name("supervisor")
            .model(selectedModel)
            .systemPrompt(dynamicInstruction)
            .methodTools(this) // expose the four @Tool methods
            .build();
        for (int attempt = 0; attempt <= maxRetries + 2; attempt++) {
            AssistantMessage response = agent.call(currentPrompt);
            ValidationResult validation = conclusionValidationAgent.validate(response.getText(), metricsQueried, eventId);
            if (validation.isPassed()) break;
            currentPrompt = buildRetryPrompt(validation.getFeedback(), response.getText());
        }
        return InvestigationResult.complete(response.getText(), suggestion);
    }
}

Four Investigation Tools and Their Design Philosophy

Tool 1 – queryLogs: Connects via WebSocket to the log platform, queries in priority order (traceId > exceptionName > endpoint > keywords), deduplicates results, and streams each batch to the LLM for asynchronous analysis.

Tool 2 – queryMetrics: Supports ten metric dimensions (qps, rt, errorRate, containerCpu, containerMemory, percentileRt, gcCount, gcRt, qpstop10, rttop10). The LLM decides which dimensions to request based on alert type. MetricService abstracts environment‑specific endpoints (e.g., csprd‑proxy for production).

Tool 3 – queryTrace: When a traceId is present, fetches the Span tree from APM, formats it as text, and lets the LLM pinpoint slow downstream dependencies or exception‑throwing nodes.

Tool 4 – queryEndpointErrors: For alerts lacking a traceId but with a known endpoint, it retrieves error logs, extracts all traceIds via regex, and analyzes each. It explicitly rejects root‑path queries ("/") to avoid meaningless scans.

Dynamic Strategy Assembly

Different services and alert types require distinct investigation paths. The system stores strategy scripts keyed by (service_name, alert_type). If no exact match exists, a built‑in fallback strategy is used. Operators can edit strategies via a front‑end UI without code changes.

instruction = ROLE + strategy(service_name, alert_type) + OUTPUT_FORMAT

Tool Timeout Isolation

External systems may be unstable. Each tool call is wrapped in ToolExecutor with its own thread pool and Future.get(timeout) handling:

public ToolResult executeWithTimeout(Tool tool, Map<String, Object> parameters) {
    Future<ToolResult> future = executor.submit(() -> tool.execute(parameters));
    try {
        return future.get(tool.getTimeoutSeconds(), TimeUnit.SECONDS);
    } catch (TimeoutException e) {
        future.cancel(true);
        return ToolResult.timedOut("Tool execution timeout");
    } catch (ExecutionException e) {
        return ToolResult.failed("Tool execution failed: " + e.getCause().getMessage());
    }
}

If a tool times out, a degraded message (e.g., "Metric query timed out, proceeding with existing evidence") is returned, allowing the LLM to continue without aborting the whole workflow.

AI Permission and Safety Guarantees

The agent operates with read‑only permissions only; it never modifies configurations, restarts services, or writes to databases. All suggestions are plain text and require human confirmation before execution.

Observability of the Investigation Process

Each investigation creates a dedicated directory (named by event ID) that stores raw alerts, system prompts, user prompts, LLM calls, tool results, and validation logs. An in‑memory EventProgressTracker updates the current stage and strategy description, which the front‑end polls every three seconds.

logs/evt-20260514153012345-001/
├── raw_alert.txt            # original alert
├── 00_RECEIVED.log          # ingestion record
├── 01_INVESTIGATION_START.log
├── 03_LLM_SYSTEM_PROMPT.log
├── 04_LLM_USER_PROMPT.log
├── 05_LLM_CALL_1.log       # first LLM call
├── 06_Tool_Call_Logs.log   # log analysis result
├── 07_LLM_CALL_2.log
├── 08_Tool_Call_Metrics.log
├── 09_LLM_FINAL_RESPONSE.log
├── 10_VALIDATION_START.log
├── 11_VALIDATION_PASS.log
└── 12_CONCLUSION.log

Real‑World Case: Gateway Timeout

An production alert from the "Efficiency Gateway" indicated a 30‑second interface timeout. The alert contained no traceId. The AI workflow performed three tool calls and one validation retry within four minutes, ultimately identifying a severe GORM‑v2 database query blockage as the root cause.

Investigation conclusion:
- Root cause: gorm‑v2 query blocked for 29.99 s, exceeding gateway 30 s timeout.
- Impact: 2 failed requests in 30 min, no cascade failure.
- Recommendations:
  1. Check slow‑query logs for missing composite index.
  2. Review gateway timeout configuration and add fallback.
  3. Evaluate business logic of multi‑condition query; consider async or batched pulls.

The total end‑to‑end time was ~4 minutes (tool calls 88 s + LLM reasoning + validation), compared with an estimated 10–20 minutes for manual handling.

Technical Challenges and Pitfalls

Environment Mapping: Production environments have many aliases ("xxprd", "prd", "生产" etc.). A unified EnvironmentProperties mapping normalizes them for both log and metric services.

LLM Rate‑Limiting: Implemented a RoundRobinChatModel that distributes API keys across events, ensuring a single key is reused throughout a given investigation to avoid context loss.

Results and Data

From 2026‑04‑21 to 2026‑05‑14, the system processed alerts across 11 services and 10+ alert types. Median investigation time dropped from ~20 minutes to 4.4 minutes. Validation pass rates were 60 % on first attempt, 38 % on second, with a 2 % fallback rate.

Future Iteration Directions

Parallelize log and metric queries across multiple agents.

Introduce a fingerprint knowledge base for sub‑second known‑issue matching.

Cross‑service correlation analysis to detect cascading failures.

Automatic remediation: evolve from "investigate + suggest" to "investigate + auto‑fix".

Vector‑based semantic retrieval for more flexible problem lookup.

The goal is not to replace operators but to offload repetitive, platform‑hopping tasks to AI, allowing engineers to focus on judgment and creative problem‑solving.