A 24/7 Product Expert: Building and Evolving a Digital Employee for the Product Center

Background and Motivation

In a typical product‑center ticket such as “product published but not visible on the client side”, engineers must follow a long, mechanical chain: receive ticket → validate basic data → check POI binding → verify engine consistency → trace upstream/downstream sync, etc. Almost every step is a deterministic data‑gathering operation, while only a few nodes require human intuition.

The goal is to embed the expert’s mental model into an autonomous Agent that, when a ticket or alarm arrives, performs all repetitive data queries and pushes the remaining judgment points to the on‑call engineer.

Why the Timing Is Right

LLM tool‑calling maturity : mainstream large models can reliably generate tool‑call specifications from a schema and reason over the returned data.

Model Context Protocol (MCP) adoption : internal core data assets are progressively exposed via MCP, allowing the Agent to discover tool metadata without custom glue code.

Agent framework abstraction : the current framework offers an OS‑like layer that integrates memory, skills, MCP, and a curator, reducing a project from “start a repo” to “configure and write knowledge”.

Early prototypes suffered from two major issues: unstable model versions caused request‑pipeline breaks, and the lack of self‑learning kept Skills static.

Choosing Hermes Agent

The selection was based on four evaluation dimensions:

Knowledge structuring & mutability : Determines whether the Agent can grow beyond a demo.

Cross‑session memory mechanism (retrieval‑style vs procedural): Determines if the Agent can retain experience across conversations.

Tool integration cost (MCP/custom extensions) : Determines how quickly the Agent can hook into the complex internal ecosystem.

Hard constraints on hallucination : Determines production‑grade trustworthiness.

Hermes Agent satisfied all dimensions, leading to four design pillars:

1. Knowledge structuring – Skills as editable programs

Skills are Markdown files with a YAML front‑matter. The Agent can patch its own Skills at runtime using the command: skill_manage action='patch' This contrasts with the common practice of keeping SOPs as static prompts or RAG snippets that require manual redeployment.

2. Memory split – Declarative vs procedural

Declarative memory ( MEMORY.md / USER.md): Stores facts such as “who I am”, environment attributes, and known parameter pitfalls.

Procedural memory (Skills): Encodes step‑by‑step processes, e.g., “how to handle a specific error”.

Only procedural memory can answer “when you see ID X next time, also check fields Y and Z”.

3. Tool integration – MCP as a first‑class citizen

Adding a new MCP server is as simple as editing config.yaml:

mcp_servers:
  commodity-mcp:   # product, SKU, shop, category, order etc.
  monitor-mcp:    # monitoring, alerts, traces
  log-mcp:        # application logs
  knowledge-mcp:  # internal knowledge base

This eliminates the need for framework‑level code changes when new internal services are added.

4. Hallucination constraints – Hard rules in SOUL.md

Every conclusion must be traceable to a concrete MCP call; if data is missing the Agent must say so, and it must ask for clarification before proceeding. These rules are baked into the Agent’s persona and enforced at the framework level.

Trade‑offs

Steep learning curve for Skill design, SOUL.md authoring, and Memory maintenance.

Relatively small ecosystem; many issues require reading source code.

Risk of “Skill drift” if poorly written Skills become entrenched and deviate from reality.

Core Capabilities Built

The Agent now supports three high‑impact scenarios:

4.1 User‑feedback diagnosis

Typical input: a screenshot of a merchant complaint such as “product not shown after publishing”. The Agent consolidates the multi‑step SOP into a single Skill call and returns a three‑part report (conclusion → recommended action → diagnostic steps). Example Skills: merchant-publish-diagnosis: handles “merchant/shop cannot publish product”. Parameters: companyId, shopId, categoryId, bProdType. item-unpresentable-diagnosis: handles “product published but not visible”. Parameters: spuId, skuId. hotel-room-match-diagnosis: handles “hotel room binding errors”. Parameters: cpId, cpName, cpSpuId. monitor-skill: handles alarm root‑cause analysis and system inspection. Parameters: appName, alarmId, time window.

Each Skill declares mandatory parameters; missing values trigger a forced clarification loop.

4.2 Alarm root‑cause analysis

Alarm flow: receive alert → extract alarmId, appName, time window → query metric trend → check recent releases → analyse logs → trace call chain → output root cause + fix suggestion. The six‑step procedure is encoded in monitor-skill and produces a uniform template for downstream ticket creation.

4.3 System health inspection

Daily health checks run as scheduled tasks, producing a three‑section report (summary → anomalies → data snapshot). The Agent injects the previous run’s output as context, enabling incremental comparison (e.g., “no anomalies today vs 3 errors yesterday”).

Self‑Evolution Mechanism

When the Agent discovers a missing or incorrect Skill step, it patches the Skill via skill_manage action='patch'. After completing a complex multi‑turn task, the Curator triggers a periodic summary, automatically writing new knowledge into MEMORY.md and USER.md. Three real cases illustrate the loop:

Case 1 – merchant‑publish‑diagnosis : Initially only a few rules lived in MEMORY.md. Repeated diagnoses caused the Agent to synthesize a full Skill file, which is now auto‑loaded for similar issues.

Case 2 – query‑statement correction : Early monitoring Skills generated wrong metric names. After repeated failures the Agent consolidated correct queries into a static file and added a hard rule to always consult it before querying.

Case 3 – service profile enrichment : Each alarm analysis updates the service’s profile (key metrics, critical APIs, known failure patterns). Subsequent analyses benefit from this accumulated knowledge, reducing time‑to‑resolution.

Key Deliverables

The project produced:

Agent framework + Gateway + multiple MCP services.

~10+ domain‑specific Skills covering merchant publishing, item visibility, hotel binding, monitoring, and log analysis.

Service profiles, standard query templates, and a dozen MEMORY entries (ID formats, business enums, etc.).

In short, the result is a “product‑center digital employee” that possesses a domain persona, real‑data constraints, self‑evolution capability, and explicit safety boundaries.

Lessons Learned

The moat lies in how well senior‑engineer expertise is structured, not in the underlying LLM.

Hard constraints that force “I don’t know” dramatically improve trustworthiness.

Skill matching must consider explicit applicability tags, not just semantic similarity.

Documentation of Skill boundaries, SOUL rules, and versioning is critical to avoid drift.

Future Roadmap

Planned enhancements include observable self‑evolution (versioned Skill history, audit logs, performance metrics), a visual knowledge‑graph dashboard, and a multi‑agent orchestration layer that splits the monolithic digital employee into specialized sub‑agents (e.g., catalog, hotel, inspection) coordinated by a routing Agent.