Building Production-Ready Ops Agents: 10 Principles, Two Architectures & Memory Design

Why Ops Agents Are Needed

The core contradiction in operations remains: system complexity grows continuously while human processing capacity is fixed. Average on-call engineers receive ~50 alerts weekly, only 2-5% requiring human intervention. 70% of SRE teams rank alert fatigue as a top-3 operational problem (PagerDuty data). A typical troubleshooting flow for "user reports payment latency" involves deterministic steps: check metrics, traces, logs, compare against baselines — essentially "tool chaining + pattern matching," which is exactly where agents excel.

Industry Landscape: 10 Production-Verified Solutions

Survey of public materials from 2024-2026 confirms 10 solutions in production verification. From these cases, 10 iron laws emerge:

Prompt for understanding, code for execution — LLMs never directly operate production systems

All write operations must be idempotent — use idempotency keys to prevent duplicate execution

Tiered safety controls — read auto / reversible write auto / irreversible needs confirmation / critical needs approval

Start with one scenario — don't try to build a general-purpose agent in one step

Define capability boundaries clearly — gracefully hand off to humans when exceeded

Full-chain observability — every tool call must be auditable

Tool-layer isolation — LLM requests tools → Agent executes safely

Vector knowledge base — RAG retrieval of historical cases

Long-term agent generalization — Microsoft converged from 50+ specialized agents to a general one, validating this path

Data flywheel — collect traces to feed model evolution

Two Architecture Paths: Deep Comparison

3.1 MCP Direct-Connection Approach

LLM (Claude/GPT/Qwen)
  │ Function Calling decides which tool
  ├── MCP: Prometheus (PromQL query)
  ├── MCP: SkyWalking (GraphQL query)
  ├── MCP: SLS (log search)
  └── MCP: GitLab (code search)
  │
  ▼ LLM consumes raw returns, reasons step by step

Each MCP server independently wraps one ops system's API. LLM uses Function Calling to decide which to call and with what parameters, then reasons over raw results.

3.2 UModel Semantic Layer Approach

LLM (Claude/GPT/Qwen)
  │ Semantic query (natural language or structured)
  ▼
UModel Unified Semantic Layer (Node+Link directed graph)
  │ Operator pushdown: aggregation/anomaly detection executed at data side
  ├── Prometheus
  ├── SkyWalking
  ├── SLS
  └── Change system
  │
  ▼ LLM receives structured summary, reasons directly

UModel is a unified modeling layer for all ops data. Agent doesn't query each system individually — UModel pre-aggregates and cross-correlates internally, so LLM asks once and gets a conclusion.

3.3 Core Difference

UModel exists not because MCP can't work, but because at scale (50+ services, 10+ ops systems), MCP direct connection's token consumption and multi-round reasoning costs explode. UModel's "data pre-aggregation + large/small model routing" compresses comprehensive cost to ~10% of MCP.

ReAct Correct Usage: Diagnosis Loops, Execution Boundaries

In ops scenarios, ReAct cannot be used everywhere. Investigation phase can use loops; execution phase runs once and must have human approval.

"Execute then check metrics then adjust" is a recipe for production incidents. ReAct loops only happen in investigation; once a write operation is reached, human checkpoint is mandatory.

LangGraph Implementation Key Points

workflow = StateGraph(InvestigationState)

# Investigation node runs ReAct internally, all tool calls read-only
workflow.add_node("investigate", investigate)

# Human approval node: LangGraph interrupt, wait for human confirmation
workflow.add_node("human_approval", human_approval)

# Execution node: only reached after human confirmation
workflow.add_conditional_edges(
    "human_approval",
    lambda s: "execute" if s["human_approved"] else END
)

Memory Management: Markdown vs Vector DB — Not Either/Or

5.1 Lessons from Claude Code

Claude Code's memory system is entirely Markdown-file based:

~/.claude/projects/<project>/memory/
├── MEMORY.md          # index, load first 200 lines each session
├── debugging.md       # topic file, read on demand
└── api-conventions.md
Project directory:
├── CLAUDE.md          # project instructions, full load each session
└── .claude/skills/    # Skill folder

Why not vector DB? Because it targets code projects with controllable knowledge scale (dozens of files). Full index load is only a few thousand tokens.

Can ops use this? Early stage, yes. Few scenarios → Markdown is simple, human-maintainable, no extra infrastructure.

5.2 Three-Layer Memory Architecture for Ops Agents

As scenarios grow, a hybrid approach is recommended:

┌────────────────────────────────────────────┐
│ L1: Short-term Memory (Working Memory)     │
│ Current diagnostic session context         │
│ Storage: LangGraph State / in-memory       │
├────────────────────────────────────────────┤
│ L2: Long-term Memory (Episodic Memory)     │
│ Historical diagnostic cases: how solved?   │
│ Storage: Vector DB (semantic search Top-3) │
├────────────────────────────────────────────┤
│ L3: Knowledge Base (Semantic Memory/Runbook)│
│ Standardized troubleshooting flows         │
│ Storage: Markdown files (structured, readable, maintainable) │
└────────────────────────────────────────────┘

5.3 Evolution Path: Standardized Flows as Skills

Referencing Claude Code's Skill mechanism, ops agent troubleshooting flows can be organized identically:

knowledge/runbooks/
├── payment-timeout.md    # Payment timeout runbook
├── oom-kill.md           # OOM Kill runbook
├── deploy-rollback.md    # Deploy rollback process
└── ...

Each Runbook structure:
  1. YAML metadata (trigger conditions, applicable services, risk level)
  2. Markdown body (step-by-step, each step linked to specific Tool Call)

LLM first sees all Runbook metadata (trigger + description), semantically matches the best one, then loads the full file to guide tool calls — exactly how Claude Code Skills operate.

Operator Pushdown: Compute at Data Side, LLM Only Gets Conclusions

"Operator pushdown" originates from database internals, applied here to observability.

Without Pushdown

Agent needs to judge "is order-service abnormal?"
  → Query Prometheus, return 10,000 metric points for order-service over 1 hour
  → Transfer to Agent local
  → Agent uses Python to compute P99 aggregation
  → LLM judges if exceeds threshold

Essence: push computation to where data lives, only transmit results.

Mining Analogy

No pushdown: haul 10 tons ore to smelter → smelt → get 1 gram gold
Pushdown: smelt on-site at mine → only ship 1 gram gold to smelter

Three-Layer Routing in UModel

This is the true source of "90% token reduction" — not just shorter single-context, but fewer total rounds + deterministic tasks handled by small models + LLM only does complex reasoning.

Zero-to-Production Tech Stack Selection

7.1 Recommended Stack

7.2 MCP Server Wrapper Example

# mcp-server-prometheus/server.py
from mcp.server import Server
from mcp.types import Tool
server = Server("prometheus-mcp")

@server.list_tools()
async def list_tools():
    return [
        Tool(
            name="prometheus_query",
            description="Execute PromQL instant query. For current service performance metrics like P99 latency, QPS, error rate",
            inputSchema={
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "PromQL query statement"}
                }
            }
        ),
        Tool(
            name="prometheus_range_query",
            description="Execute PromQL range query. For time-series data over a period, helps identify when anomaly started",
            inputSchema={
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "start": {"type": "string"},
                    "end": {"type": "string"},
                    "step": {"type": "string"}
                }
            }
        )
    ]

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "prometheus_query":
        response = requests.get(
            f"{PROMETHEUS_URL}/api/v1/query",
            params={"query": arguments["query"]}
        )
        return response.json()

7.3 Runbook Markdown Example

---
trigger: "User reports payment latency"
services: ["order-service", "payment-service"]
risk_level: "MEDIUM"
tools: ["prometheus_query", "skywalking_trace", "sls_search"]
---
# Payment Timeout Troubleshooting Runbook
## Step 1: Confirm anomaly scope
**Tool**: `prometheus_query`
**Query**: `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="order-service"}[5m]))`
**Judgment**: P99 > 2s → continue Step 2
## Step 2: Locate slow call segment
**Tool**: `skywalking_trace`
**Action**: Get slow traces for order-service
**Judgment**: Which span > 2s → record traceId → continue Step 3
## Step 3: Deep-dive logs
**Tool**: `sls_search`
**Query**: `traceId: {from previous} AND level: ERROR`
**Judgment**:
- Found timeout logs → downstream issue → check recent downstream deploy
- No timeout logs → check code logic → continue Step 4
## Common Root Cause Quick Reference
| Root Cause | Frequency | Fix Action | Est. Recovery |
|------------|-----------|------------|---------------|
| Bank API timeout | 40% | Enable circuit breaker + notify counterpart | 10min |
| DB slow query | 30% | Index optimization | 30min |
| Gateway timeout config | 15% | Adjust parameter | 5min |