One‑Click AI Knowledge Base Solves Massive Document Search for K8s Fault Root‑Cause Analysis

In Kubernetes production environments, fault diagnosis heavily depends on engineers' experience, while internal SOPs, troubleshooting manuals, and post‑mortem documents are scattered across wikis and word files, making it impossible to consult them during an incident.

The article presents a self‑developed K8s‑RAG‑AIOps visual troubleshooting tool that combines a local vector knowledge base (bge‑small‑zh) with the DeepSeek‑v4‑pro large model to automatically retrieve relevant internal documents and generate a complete fault‑handling report containing executable commands.

Tool architecture consists of four layers:

SSH remote collection layer: uses paramiko to connect to the K8s control node and run kubectl commands, gathering pod describe, container logs, events, and resource usage.

Local RAG private knowledge‑base layer: loads embeddings and document chunks offline, builds a vector store of internal SOPs, and performs similarity search without any network access.

Large‑model inference layer: feeds the raw cluster data together with the retrieved knowledge snippets to DeepSeek‑v4‑pro, forcing the model to prioritize enterprise‑specific guidelines.

GUI visualization layer: a tkinter desktop app where users fill in SSH credentials and DeepSeek API key, then trigger diagnosis with a single click.

Capability 1 – One‑click pod fault diagnosis : after entering the control‑node IP, credentials, and selecting a namespace and problematic pod, the tool executes the SSH commands, performs RAG retrieval, sends a prompt to the model, and outputs a six‑section report covering overview, risk assessment, root‑cause analysis, emergency recovery commands, validation criteria, and long‑term optimization suggestions.

Capability 2 – Cluster node resource & scaling analysis : the tool runs kubectl top nodes to obtain real‑time CPU/memory usage, retrieves internal scaling policies via RAG, and the model produces a resource summary, risk judgment, vertical/horizontal scaling recommendations, eviction commands, and long‑term optimization strategies.

Core code snippets illustrate the offline RAG loading configuration (environment variables to block internet access, loading embeddings.npy and chunks.pkl, initializing SentenceTransformer), the SSH command execution function, and the RAG retrieval function that encodes the fault description, computes cosine similarity, and returns the top matching document fragments.

Realized value includes consolidating years of SOPs into a private vector store to preserve senior engineers' knowledge, accelerating troubleshooting by eliminating manual document search, lowering the expertise barrier for junior SREs, and ensuring data never leaves the intranet, which satisfies security requirements for enterprises, operators, and railways.

Future directions aim to integrate Prometheus alerts for automatic trigger, ingest EFK logs for richer context, and expand the fault‑case library so that historical incidents can be directly reused.