AgentScope Bible 02: 7 Onion Middleware Slots, 3-Layer Architecture & ReAct Loop Control

Why Traditional Pre/Post Hooks Fail for Streaming Agents

The article opens with an interview question: why does a simple linear callback model (pre/post hooks) not work for ReAct middleware slots? The author explains that pre/post hooks assume synchronous, one-shot request-response with only start and end observation points. Agent streaming breaks three premises: (1) output is incremental delta chunks, not a single result; (2) users can cancel generation at any time, causing post hooks to never run; (3) Human-in-the-loop (HITL) can pause execution for minutes or hours, which pre/post models cannot handle. Additionally, middleware must process each streaming event individually (e.g., sensitive-word filtering per chunk, audit per tool call).

Onion Middleware Contract: Async Generator Wrapper

AgentScope redefines middleware as a function that receives an async generator and returns a new async generator, with a next_handler parameter to invoke the inner layer. The canonical template:

async def on_reply(self, agent, kwargs, next_handler):
    # pre logic
    span = tracer.start_span("agent.reply")
    try:
        async for event in next_handler(kwargs):
            # per-event processing: modify, intercept, short-circuit
            yield event
    finally:
        # cleanup runs even on cancellation
        span.end()

Key points: next_handler is a closure (not yet executed), giving the middleware control over when/if to delegate. The async for loop consumes inner events one by one, enabling per-event transformation. The finally block guarantees cleanup on normal completion, cancellation ( GeneratorExit), or error.

Comparison: Pre/Post vs Onion Model

Execution Model : Pre/Post uses two separate loops (all pre, then all post); Onion uses recursive nested async generators, stack model.

Order Semantics : Pre/Post are FIFO for both pre and post; Onion chain is first-in-last-out.

Cancellation : Pre/Post loses post hooks, resources leak; Onion triggers GeneratorExit, finally runs.

Event Granularity : Pre/Post only sees final result; Onion intercepts every streaming event.

Representative Slots : Pre/Post uses global callback list; Onion provides on_reply, on_model_call, on_acting, etc.

7 Fixed Slots + 3-Layer Onion Architecture

AgentScope defines 7 fixed extension points (6 onion hooks + 1 transformer hook):

Onion Hooks (carry next_handler, control execution flow): on_reply – wraps entire ReAct loop on_reasoning – LLM thinking phase on_acting – tool invocation on_check_permission – permission checks on_model_call – low-level LLM call on_compress_context – context compression

Transformer Hook (no next_handler, pure text pipeline): on_system_prompt – modifies system prompt string

Middleware registration uses "implement-to-register": the framework compares method references between base class and subclass; if different, the hook is auto-registered. Base class methods include a dummy yield to ensure return type is AsyncGenerator for type consistency.

Three-layer isolation by data visibility:

ToolMiddleware – wraps single tool call; sees only that tool's input/output.

Agent-level MiddlewareBase – single agent/session lifecycle; sees full reasoning messages, state.

App-level Business Middleware – cross-agent, cross-session; orchestration layer (rate limiting, team SOP).

Recursive Scheduling: execute_chain

The onion chain is built via recursive closure construction, not a simple loop. Simplified pseudo-code:

async def execute_chain(mw_list, idx, agent, kwargs):
    if idx >= len(mw_list):
        return agent.raw_reply_stream(kwargs)  # terminal business logic
    current_mw = mw_list[idx]
    async def next_handler(inner_kwargs):
        return execute_chain(mw_list, idx + 1, agent, inner_kwargs)
    return current_mw.on_reply(agent, kwargs, next_handler)

Each middleware receives a next_handler closure representing the rest of the chain. It decides whether to call it, when to call it, and how to process the returned async generator. This yields stack-like first-in-last-out behavior automatically.

Three Core Middleware Patterns

Short-circuit – skip next_handler, emit a terminal event (e.g., Event(type="deny")). Unlike throwing exceptions, the stream closes gracefully; upstream consumers handle the event normally.

Event Rewriting – delegate to next_handler, then modify each event in the async for loop (e.g., sensitive data masking on delta chunks).

Parameter Injection – enrich kwargs before delegating (e.g., add token budget). Note: kwargs flows downward only; modifications don't propagate back up.

ReAct Loop Control via ReplyEndEvent

Because ReAct is a multi-turn loop, the framework cannot rely on generator exhaustion to end a round. Instead, it defines a ReplyEndEvent signal and a boolean flag _receive_reply_end. The main loop:

async def react_loop(agent, kwargs):
    agent._receive_reply_end = False
    while not agent._receive_reply_end:
        async for event in agent.run_reply_chain(kwargs):
            if isinstance(event, ReplyEndEvent):
                agent._receive_reply_end = True
            yield event

If any middleware swallows the ReplyEndEvent (does not yield it), the flag stays false and the loop restarts, enabling quality checks (citations, tool errors, reasoning drift) to force another reasoning round without touching core loop code. This delegates loop continuation decisions to cross-cutting middleware, adhering to Open/Closed Principle.

System Prompt Transformer Hook

on_system_prompt

is a pure text pipeline (first-in-first-out, no next_handler). Each middleware receives the current prompt string and context, returns a modified string. Best practice: append to dedicated marker blocks (e.g., <memory>...</memory>) rather than overwriting the whole string, to avoid losing prior middleware contributions.

Tool Offload & State Consistency in on_acting

Slow I/O tools can be offloaded to background tasks. The framework uses an is_state_injected flag on tools: False – tool receives a snapshot; safe to offload. True – tool holds live agent.state reference; offload forbidden (forced synchronous) to avoid data races.

For write-back scenarios, use versioned CAS updates or session lease locks; never deep-copy entire state back, as it would overwrite concurrent main-loop changes.

Async Generator Safe Closure with aclosing

Nested async generators require each layer to propagate cancellation signals. The correct pattern wraps the inner stream with contextlib.aclosing:

from contextlib import aclosing

async def on_reply(self, agent, kwargs, next_handler):
    async with aclosing(next_handler(kwargs)) as stream:
        try:
            async for event in stream:
                yield event
        finally:
            self.cleanup()  # runs on normal exit or cancellation

aclosing

ensures that when the outer layer exits (via GeneratorExit), it calls aclose() on the inner generator, which propagates down the chain. Without this, inner spans, DB connections, or file handles leak silently. The bottom-most business generator uses its own try/finally to clean up resources when GeneratorExit arrives.