Engineering a Multi‑Agent System: Architecture, Stability, and Observability Lessons

01 Why Multiple Agents

Attempting a single "super Agent" to handle intent understanding, data fetching, gap calculation, proposal generation, review, and downstream write‑back quickly exposed four problems:

Context explosion : a single request must supply user profile, historical data, product catalog, rule base, and logs, which cannot fit into one prompt and leads to attention drift.

Failure hard to locate : when an LLM answer is wrong, it is unclear whether the error originates from data retrieval, reasoning, or formatting.

Deterministic steps broken : tasks with clear rules, such as content review, become uncontrollable if delegated to a free‑form LLM.

Cost uncontrolled : every step consumes large‑model tokens, causing rapid cost growth.

Conclusion : split the workflow into nodes, let LLMs handle only understanding and generation, and assign deterministic logic to code and rules.

02 Agent Architecture: 3 LLM Agents + 1 Skill

The final architecture consists of three LLM agents arranged in a linear pipeline, followed by a deterministic skill that performs pure code execution.

The core principle is "the intelligent part stays intelligent, the deterministic part stays deterministic".

Requirement Analyst : LLM parses user intent, structures a profile, and orchestrates tools to collect data, compute gaps, match products, and match benefits—all within one agent.

Solution Designer : LLM generates proposal text and extracts highlights, preserving the creative portion.

Reviewer : L1 keyword rules (millisecond‑level, deterministic) plus L2 LLM semantic review; anything that can be decided by rules never reaches the LLM.

Synchronous Execution (Skill) : pure code flow for tagging, task creation, write‑back, and push; no LLM involved.

03 Orchestration Layer: LangGraph State Machine + Shared State Table

LangGraph was chosen because it provides a true state machine. Nodes exchange an explicit State (a TypedDict) instead of stuffing context into prompts and hoping the model remembers it.

Using an explicit state makes failures instantly visible: the state shows what was read and written, dramatically reducing debugging effort.

04 Tool Layer: MCP (Model Context Protocol)

Agents need actionable capabilities. MCP wraps external services, exposing five servers and 25 tools (e.g., data.get_user_profile, channel.push_notice) with module‑prefix naming.

Tool name prefixes : data.* and channel.* make names readable and avoid cross‑server naming conflicts.

Unified registry : a @mcp_tool decorator plus a registry enables plug‑and‑play tools without agents needing to know implementation details.

Result desensitization : phone numbers and ID numbers are masked before entering prompts or logs.

Write‑operation auth : read operations are open; write operations require JWT to prevent unauthenticated calls.

05 Stability Engineering: Circuit Breaker, Retry, Degradation

The biggest threat is an LLM suddenly misbehaving, causing the whole pipeline to collapse. Each agent is wrapped with run_with_fallback, which implements three mechanisms.

Circuit breaker : after 5 consecutive failures or a 50% failure rate within a 30 s window, the node trips and enters a cooldown probe phase, preventing a single glitch from taking down the entire chain.

Exponential backoff retry : for timeout or network errors, retries occur with delays 1 s → 5 s → 20 s, up to three attempts.

Degradation : when an LLM becomes completely unavailable, fallback to rule‑based parsing, template responses, and cached data. The result remains usable but never returns a 500 error.

The review step uses a 12 s timeout with at most one retry, deliberately more aggressive than other agents because a stalled review is worse than a degraded rule‑based response.

06 Memory System: Giving Agents "Memory"

A three‑level memory hierarchy was built:

L1 Session Memory : the shared State provides intra‑request state sharing.

L2 User Memory : persisted in PostgreSQL with Redis caching. After generating a proposal, update_customer_memory accumulates preferences (budget, needs, product likes) and stores the conversation for the next session.

L3 Knowledge Memory : ChromaDB vector store holds product, service, review‑rule, and historical proposal embeddings, enabling semantic retrieval instead of keyword matching.

Any successful write must invalidate the corresponding cache; otherwise stale profiles are served. A connection‑pool lock and failure‑cooldown were added to avoid retry storms.

07 Observability: End‑to‑End TraceID

A global X-Trace-Id is generated for each request and propagated via contextvars across all nodes.

All logs automatically include the TraceID in structured JSON.

A query endpoint /trace/{trace_id} lets developers replay the full execution chain.

The frontend SSE stream also carries the TraceID, so users can copy an ID when reporting issues.

08 Streaming Experience: Making the "Thinking Process" Visible

Server‑Sent Events (SSE) push each agent’s status to the frontend in real time:

agent_start → agent_complete → compliance_result → final_result

This turns a generic loading spinner into a meaningful view of which agent is currently working, and the UI presents each node as a role‑named, icon‑decorated workbench.

09 Pitfalls Encountered

Pitfall 1: More agents ≠ smarter – assigning an agent to every capability inflates failure surface, token usage, and latency. Split only when a step truly requires language ability; otherwise implement it as code.

Pitfall 2: State passed via dialogue – early attempts let agents reference each other’s full context, causing token explosion and "drift". Switching to a shared State limited each node to the fields it needed.

Pitfall 3: Giving deterministic logic to LLMs – content review and system write‑back were first handled by LLMs, leading to uncontrollable outcomes. The final rule: if a rule can decide, use the rule; if a process can be coded, use code; reserve LLMs for understanding.

Conclusion

Architecture : state‑machine orchestration plus a tool layer gives clear, replaceable agent responsibilities.

Stability : circuit breaker, retry, and degradation turn LLM instability into manageable risk.

Engineering : full‑chain TraceID, SSE streaming, and layered memory provide observability and a smooth user experience.

Cognition : more agents are not always better, and deterministic logic must stay out of LLMs.