HolmesGPT: AI-Powered SRE Agent That Correlates Alerts, Logs, Metrics for Root Cause Analysis

What Is HolmesGPT

HolmesGPT is an open-source SRE agent positioned to investigate production incidents and help locate root causes. It supports any Kubernetes cluster and can connect to virtual machines, cloud services, databases, SaaS platforms, and various observability systems (currently a CNCF Sandbox project). From an operations perspective, it functions as a troubleshooting investigation assistant — not a tool that presses restart buttons, but a companion that answers questions such as:

Why is this service unhealthy?
Why won't this Pod start?
What Prometheus alerts are currently firing?
What anomalies exist in this namespace?

It then queries the data sources it has access to: Kubernetes resource status, Pod logs, Prometheus metrics, Grafana dashboards, Loki logs, Datadog, AlertManager, PagerDuty, Jira, GitHub, GitLab, Jenkins, and more. HolmesGPT's direction is to correlate alerts, metrics, logs, resource state, and change information around a single incident — mirroring real-world operations workflows.

The Problem It Solves: The First 10–20 Minutes

HolmesGPT addresses not "how to fix" but "how to investigate." Most production incidents consume the most time in the first 10–20 minutes: which service to check first, application vs. Kubernetes scheduling issue, new version vs. dependency failure, traffic spike vs. resource limit. These questions are not hard individually, but scattered information causes operators to chase dead ends — especially during on-call at night when fatigue drives people to their most familiar tools (logs, metrics, kubectl describe) rather than the shortest path.

HolmesGPT's value: it acts as an agent that calls tools to query resources, logs, metrics, and alerts, then produces an investigation conclusion with evidence. The conclusion may not be 100% correct, but listing the evidence already saves time. For example, it might report:

A Deployment's Pod is in CrashLoopBackOff;

Previous Pod logs show database connection timeouts;

At the same time, Prometheus shows database connection pool saturation;

The most recent deployment changed connection pool parameters.

Getting Started Locally

Installation (macOS/Linux via Homebrew)

brew tap robusta-dev/homebrew-holmesgpt
brew install holmesgpt
holmes ask --help

Use your local kubeconfig for initial testing. Verify the current context: kubectl config current-context Configure the model API. HolmesGPT uses LiteLLM to call LLMs and supports OpenAI-compatible endpoints (the model must support Function Calling).

Run a Test

After configuration, ask a concrete question:

holmes ask "请用中文检查当前集群中状态异常的 Pod，说明异常原因，并列出判断依据。"

For a guided walkthrough, create a problematic test Pod:

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml

Then ask HolmesGPT to investigate:

holmes ask "请用中文调查 user-profile-import Pod 为什么无法调度，并给出证据和建议的处理步骤。"

Once this runs, you understand its working mode.

Three Production Adoption Patterns

1. Read-Only Investigation

Grant a read-only kubeconfig or ServiceAccount so HolmesGPT can only view resources, logs, and metrics — not modify anything. Run first in test or staging clusters. The goal is not pretty answers but stable detection of real anomalies, avoiding hallucinations or mislabeling unrelated data as root cause.

2. Alert-Assisted Analysis

When Alertmanager fires a service-unavailable alert, don't ask HolmesGPT to fix it. Instead, prompt it with fixed investigative questions:

What anomalies do you see?
Which data sources do they come from?
What is the most likely direction?
What specific points should a human check next?

This prompt style constrains the tool to investigation and recommendation, which is safer for production.

3. Postmortem Evidence Preservation

After incidents, teams often realize they didn't preserve scene evidence: Pod state, restarts, error logs, Prometheus metric spikes. If HolmesGPT's investigation process is retained, it becomes part of the postmortem record.

How It Differs from Traditional Scripts

Ops teams have long written inspection scripts that check Pod status, restart counts, PVCs, nodes, certificate expiry, disk, quotas. Scripts only follow predefined rules; they miss anything not explicitly coded. HolmesGPT adds an agentic layer: it decides what to query next based on the question — e.g., for an unhealthy Pod it may check status, then events, then logs, then the Deployment, then Prometheus metrics if connected. This resembles an experienced operator's troubleshooting flow, though quality depends on tool configuration, data quality, and model capability. It does not replace scripts; a better relationship is scripts for stable periodic inspections, HolmesGPT for ad-hoc investigations.

Critical Boundaries for Production Use

No write permissions initially. HolmesGPT has Kubernetes remediation capabilities (scale, rollback, modify resources), but these should not be enabled in production upfront. Start read-only. Only after the team trusts its output quality, audit process, and rollback procedures should execution be discussed — never reverse this order.

Data egress and sensitive information. While the Kubernetes toolset excludes secrets by default, logs may contain tokens, internal domains, user data, database errors, API paths. If data egress policies exist, consider local/on-premise models or redaction. Do not ship production logs wholesale to external models just because the tool is convenient.

AI provides investigation conclusions, not final root cause attribution. It can say "I observe these phenomena, so I suspect this direction." But definitive root cause requires evidence closure: deployment records, metric inflection points, log timelines, rollback verification. Missing any piece means the conclusion should not be stated as absolute.

Don't let juniors bypass fundamentals. The tool helps newcomers understand the scene faster, but it should not replace learning Kubernetes, Prometheus, logs, and release processes. Best practice: read HolmesGPT's conclusion, then verify with kubectl describe, kubectl logs, PromQL, and raw logs. Verification builds skill faster than passive reading.

Final Assessment

HolmesGPT is more like an incident-investigation-focused AI ops assistant compared to K8sGPT, which centers on Kubernetes resource diagnostics. HolmesGPT aims higher: unify alerts, logs, metrics, resource state, tickets, and code platforms into a single investigation. This direction is worth watching. In production, the stance remains:

Let it read first.
Then let it explain.
Then let it suggest.
Only then discuss whether it can execute.

If you already run Kubernetes, Prometheus, Loki, or Grafana and want to test whether AI can genuinely assist in ops troubleshooting, HolmesGPT is a logical next step after K8sGPT. Don't expect it to handle incidents end-to-end immediately. Let it clarify the scene first — that alone saves significant time.

Repository:

github.com/HolmesGPT/holmesgpt