Building an AI-Powered OnCall Agent: RAG and Plan‑Execute‑Replan in Practice

System Overview

Intelligent OnCall Agent is an enterprise‑level AI automation assistant that integrates three core agents—knowledge‑base, conversational, and operations—to answer questions and diagnose incidents automatically, reducing on‑call labor costs and improving team efficiency.

Environment Setup

Clone the repository: git clone https://github.com/gofish2020/OncallAgent.git Start Docker services: cd ./manifest/docker && docker-compose up -d Copy config.example.yaml to config.yaml and fill in required API keys.

Install Ollama and pull the nomic-embed-text embedding model (or another Ollama embedding model). Download Ollama from https://ollama.com/download.

Backend Architecture

Overall Layered Design

The entry point is main.go, which creates a GoFrame server, registers CORS and response‑formatting middleware, binds the chat controller, and listens on port 6872.

Initialize GoFrame context.

Set up middleware (CORS, response format).

Register API route group /api.

Run HTTP server on http://localhost:6872.

API Endpoints

/api/chat

(POST) – quick dialogue, request ChatReq, response ChatRes. /api/chat_stream (POST) – streaming dialogue, request ChatStreamReq, response ChatStreamRes. /api/upload (POST) – file upload, request FileUploadReq, response FileUploadRes. /api/ai_ops (POST) – AI operations analysis, request AIOpsReq, response AIOpsRes.

Control Layer Implementation

Chat – Quick Dialogue Interface

Request flow: ChatReq(Id, Question) → build UserMessage → construct chat agent via chat_pipeline.BuildChatAgent → invoke agent → store user and AI messages in session memory → return ChatRes(Answer).

func (c *ControllerV1) Chat(ctx context.Context, req *v1.ChatReq) (res *v1.ChatRes, err error) {
    userMessage := &chat_pipeline.UserMessage{
        ID: req.Id,
        Query: req.Question,
        History: mem.GetSimpleMemory(req.Id).GetMessages(),
    }
    runner, err := chat_pipeline.BuildChatAgent(ctx)
    out, err := runner.Invoke(ctx, userMessage)
    mem.GetSimpleMemory(req.Id).SetMessages(schema.UserMessage(req.Question))
    mem.GetSimpleMemory(req.Id).SetMessages(schema.SystemMessage(out.Content))
    return &v1.ChatRes{Answer: out.Content}, nil
}

ChatStream – Streaming Dialogue Interface

Request flow: ChatStreamReq(Id, Question) → create SSE client → build UserMessage → invoke agent.Stream() → send each chunk to the client → after the stream ends, persist the full conversation in memory.

AIOps – AI Operations Analysis Interface

Execution flow: receive AIOpsReq, call plan_execute_replan.BuildPlanAgent to create a planner (DeepSeek V3), executor (tool set), and replanner (up to 20 iterations). The planner generates a plan, the executor runs tools such as query_prometheus_alerts, query_internal_docs, get_current_time, and query_log, and the replanner revises the plan based on intermediate results. The final AIOpsRes contains a structured alert‑analysis report.

func BuildPlanAgent(ctx, query string) (string, []string, error) {
    planner := NewPlanner(ctx)
    executor := NewExecutor(ctx)
    replanner := NewRePlanAgent(ctx)
    planExecuteAgent := planexecute.New(ctx, &planexecute.Config{
        Planner: planner,
        Executor: executor,
        Replanner: replanner,
        MaxIterations: 20,
    })
    result := runner.Query(ctx, query)
    return finalMessage, detailSteps, nil
}

AI Agent Pipelines

Chat Pipeline

The pipeline consists of nodes InputToRag, InputToChat, MilvusRetriever, ChatTemplate, and ReactAgent. Edges define data flow from start to end, ensuring that query strings are retrieved from Milvus, formatted into prompts, and fed to the LLM.

func BuildChatAgent(ctx context.Context) compose.Runnable[*UserMessage, *schema.Message] {
    g := compose.NewGraph[*UserMessage, *schema.Message]()
    g.AddLambdaNode("InputToRag", newInputToRagLambda)
    g.AddChatTemplateNode("ChatTemplate", chatTemplate)
    g.AddLambdaNode("ReactAgent", reactAgent)
    g.AddRetrieverNode("MilvusRetriever", milvusRetriever, compose.WithOutputKey("documents"))
    g.AddLambdaNode("InputToChat", newInputToChatLambda)
    g.AddEdge(compose.START, "InputToRag")
    g.AddEdge(compose.START, "InputToChat")
    g.AddEdge("InputToRag", "MilvusRetriever")
    g.AddEdge("MilvusRetriever", "ChatTemplate")
    g.AddEdge("InputToChat", "ChatTemplate")
    g.AddEdge("ChatTemplate", "ReactAgent")
    g.AddEdge("ReactAgent", compose.END)
    return g.Compile(ctx)
}

Knowledge Index Pipeline

Four steps process documents into a vector store: load Markdown files, split by headings/paragraphs, embed text with an Ollama embedding model, and store vectors with metadata in Milvus.

Plan‑Execute‑Replan Pipeline

Implements a loop of planning (DeepSeek V3 reasoning), execution (tool calls), and replanning (up to 20 iterations) to handle complex operational tasks.

Tool System

query_internal_docs

– RAG search of internal documentation. query_prometheus_alerts – fetch active Prometheus alerts. get_current_time – obtain the current timestamp. query_log – retrieve logs via Tencent Cloud CLS (MCP protocol). query_metrics_alerts – fetch metric‑based alerts. MySQL CRUD – execute arbitrary SQL statements with user confirmation.

Vector Models and Configuration

The system supports two embedding models defined in config.yaml:

Alibaba Baichuan text-embedding-v4 (requires API key and endpoint https://dashscope.aliyuncs.com/compatible-mode/v1).

Ollama local model nomic-embed-text (endpoint http://localhost:11434).

Key Technical Points

RAG (Retrieval‑Augmented Generation) – retrieve relevant documents from Milvus, augment the prompt, and generate accurate answers.

Plan‑Execute‑Replan – decompose complex tasks into planning, execution, and dynamic replanning, offering fault tolerance and adaptability for ops automation.

Tool Integration – Eino framework enables function calling; tool results are automatically injected into LLM context for seamless external system interaction.

Session Management – per‑session memory cache with a sliding window of six messages, thread‑safe via mutex, preserving user‑AI message pairs.

Architecture Diagram