When a Tool Call Ends: Observation Layer as the Quality‑Check Station for AI Agents

Problem Scenario: The Main Loop Runs but Answers Remain Unreliable

Even after assembling modules 1‑5 to form the ReAct loop (brain decision → tool execution → observation write‑back → next decision), real‑world user requests expose several shortcomings in the observation and closing stages:

Bad observation treated as truth : a tool returns a 500‑page HTML or {"ok": false} but the model ignores it and proceeds as if the query succeeded.

Errors not utilized : after a timeout the model retries the same arguments three times, receiving CITY_NOT_FOUND and then switches to an even more misspelled city.

Infinite loops : five consecutive steps like get_weather(city=上嗨) exhaust max_steps, leaving the user only with a token bill.

Premature termination : when the user asks for a Shanghai‑vs‑Beijing comparison plus dressing advice, the model stops after Shanghai and fabricates the rest.

Unconstrained closing : final answers lack tool source citations, miss required JSON fields, and leave the user unaware of partial completion.

The root cause is not a "dumb model" but the absence of an independent observation layer that defines the shape of tool returns and how they are written back into the context.

Position of the Observation Layer in an Agent

Within each loop, after the tool executor finishes, the observation layer sits between Execute and the next Decide . It receives the raw result, performs serialize / enrich / classify, and writes the processed observation back via ContextManager. For branches without tool_calls, it also performs closing validation and formatting before returning to the user.

Core Concept 1: Observation – Turning Tool Results into the Next Input

An Observation is not the raw executor dump; it is the result after the observation layer processes it and writes a string (usually JSON) into the next round's brain.

Three Consumers, One Formatting Pass

Model (next Decide) needs a short, structured, success‑or‑failure clear payload: ok + data or error, preserving key fields.

Context layer (module 2) requires token‑efficient, atomic pairs; the observation layer performs serialize() to shrink payloads, storing large payloads only in trace.

Orchestration / Observation (module 5/11) needs classified fields such as error.code, latency, and retry flags.

Write‑Back Rules

N calls → N observations , aligning tool_call_id strictly.

Partial failure : if three parallel calls produce one timeout, write back the two successes unchanged and the failed one as ok: false – do not discard the whole batch.

Never masquerade success : HTTP 200 with an error page must be mapped to ok: false.

Optional orchestration hint : e.g. {"suggested_action": "retry_with_pinyin"} can guide the model without replacing its reasoning.

Enriching Observations

Sometimes a very short natural‑language summary is wrapped around the JSON content to lower parsing failures, e.g.:

{
  "ok": true,
  "data": {"city": "上海", "temp_c": 28},
  "_hint": "已成功获取上海实时天气，可继续查询北京"
}

Trade‑off: the hint reduces misinterpretation but repeats data and adds tokens, so enable it only for complex multi‑call scenarios or less‑experienced models.

Core Concept 2: Error Grading & Recovery – Retry, Switch Tool, Degrade

Module 3 already attaches error.code and retryable to the observation. The observation layer builds a recovery‑strategy table used by the orchestration layer’s Check phase; the model is not left to decide on every error.

Error Levels

L0 (retriable) – codes like TIMEOUT, RATE_LIMIT; same tool & args, exponential back‑off, auto‑retry ≤ N times, then write final observation.

L1 (corrections) – INVALID_ARGS, CITY_NOT_FOUND; change parameters, write back error, no auto‑retry, let ReAct handle.

L2 (switch path) – UPSTREAM_DOWN or main tool unavailable; suggest fallback tools, possibly trigger a Replanner.

L3 (irrecoverable) – FORBIDDEN, UNKNOWN_TOOL; stop the path, either degrade or END.

L4 (business rejection) – POLICY_VIOLATION; explicitly refuse with reason or hand over to HITL.

Three Recovery Modes

Automatic retry (L0): orchestration retries the executor within the same step, counting retries in the trace.

Model‑driven recovery (L1/L2): after observation write‑back, the model may adjust arguments, switch tools, or invoke a Replanner.

Orchestration degradation (L3, max_steps, repeat_loop): observation layer produces a degraded final template describing what was completed and what is missing.

Iron rule: errors with retryable: false must never be silently retried by the orchestration layer, otherwise the model never sees the failure and loops risk increases.

Core Concept 3: Self‑Critique – When to Reflect and How to Avoid Infinite Loops

Self‑critique (or Reflection) is inserted by the observation layer at key nodes to compare the current state against the user goal and the Definition of Done.

Trigger Points

In‑step critique : after a batch of observations is written back and before the next Decide; checks questions like “Has the sub‑question been answered? Should we switch tools?”

Closing critique : when the model produces no tool_calls and is about to emit the final answer; checks “Did we satisfy all user requirements? Are any facts still missing?”

In‑step critique is costly; enable it only for high‑risk or multi‑sub‑task scenarios. Limit critique rounds to 1‑2 per task to prevent “reflection‑in‑the‑loop” runaway.

Implementation Shape

{
  "satisfied": false,
  "gaps": ["尚未查询北京天气", "未给出穿衣建议"],
  "next_action": "continue_react"
}

If satisfied is false, the orchestration injects a system message such as “【质检】当前尚未满足：尚未查询北京天气；请继续执行计划。” and forces a CONTINUE step.

Core Concept 4: Closing and Output Constraints – Defining When the Task Is Truly Done

Absence of tool_calls does not equal task completion. The observation layer must run a Definition of Done check and format the output.

Definition of Done

用户请求：查上海和北京天气，对比并给穿衣建议。
Done：
 - [ ] 已获取上海天气（来自 tool）
 - [ ] 已获取北京天气（来自 tool）
 - [ ] 回答中包含两城对比
 - [ ] 回答中包含穿衣建议

The check scans recent tool results for both cities and verifies that the final text contains comparison keywords, blocking obvious premature termination.

Output Constraints

Format : Markdown / JSON schema / fixed fields (e.g., customer‑service ticket).

Source citation : facts must be attributed to the originating tool or RAG chunk, e.g., “上海 28°C（来源：get_weather）”.

Refusal : when all tools fail and no reliable knowledge exists, explicitly state inability instead of fabricating.

PII / length : apply pre‑filter truncation before module 10.

For high‑risk scenarios, the model first outputs JSON; the observation layer validates and renders it. If parsing fails, a single repair call is attempted; persistent failure falls back to a plain‑text degraded answer with error explanation.

Core Concept 5: Repeat‑Loop Detection – The Observation Layer Provides the Brakes

During the Check phase (module 5), the observation layer calls _detect_repeat_loop. It must explicitly define detection rules to keep project‑wide magic numbers consistent.

Typical Repetition Patterns

Same tool, same args : identical (name, args_hash) for N consecutive steps → break and emit degraded final.

Same error, same args : same error.code + args for M steps → stop retry, suggest changing city or tool.

No progress : step count increases but Done checklist gains no new ticks → trigger Replanner or degrade.

Critique spin‑loop : two consecutive gap lists are identical → stop CONTINUE and degrade.

Before comparing JSON signatures, canonicalize (sort keys, trim strings) to avoid false negatives such as "上海" vs " 上海 ".

Pseudocode: Minimal ObservationHandler Skeleton

class ObservationHandler:
    """观察层：格式化、分类、恢复建议、Done 检查、降级文案"""

    def __init__(self, config, done_checker, critique=None):
        self.max_auto_retry = config.max_auto_retry_per_call
        self.repeat_threshold = config.repeat_same_call_threshold
        self.critique = critique  # optional LLM or rule
        self.done_checker = done_checker

    def process_batch(self, calls, raw_results, ctx, trace):
        """Execute 之后、append_tool_results 之前"""
        observations = []
        for call, raw in zip(calls, raw_results):
            obs = self._normalize(raw)
            if not obs.ok and obs.error.retryable:
                obs = self._auto_retry(call, obs, trace)
            observations.append(obs)
        self._record_for_repeat_detection(calls, observations, ctx)
        return [self.serialize(o) for o in observations]

    def after_observe(self, ctx, user_goal):
        """一批 tool results 写回后：步内反思（可选）"""
        if self._any_failed(ctx.last_tool_results()) or self._needs_step_critique(user_goal):
            verdict = self.critique.step_check(ctx, user_goal)
            if not verdict.satisfied:
                ctx.inject_system(f"【质检】{verdict.gaps_summary()}")
                return StepOutcome.CONTINUE
        return StepOutcome.CONTINUE

    def before_finalize(self, ctx, draft_final, user_goal):
        """无 tool_calls 时：收尾反思 + Done"""
        if self.critique:
            v = self.critique.final_check(ctx, draft_final, user_goal)
            if not v.satisfied:
                return FinalizeOutcome.REJECT, v.gaps
        if not self.done_checker.passes(ctx, user_goal):
            return FinalizeOutcome.REJECT, self.done_checker.missing_items()
        return FinalizeOutcome.ACCEPT, self.format_final(draft_final, ctx)

    def detect_repeat_loop(self, ctx):
        recent = ctx.recent_tool_signatures(limit=self.repeat_threshold + 1)
        if len(recent) >= self.repeat_threshold:
            if len(set(recent)) == 1:
                return True, "same_call_repeat"
        return False, None

    def build_degraded_final(self, ctx, reason, user_goal):
        done_part = self.done_checker.summarize_completed(ctx, user_goal)
        missing = self.done_checker.missing_items(ctx, user_goal)
        return DegradedFinal(text=render_template(done_part, missing, reason), reason=reason)

The handler connects to module 3 ( _normalize producing ok / error), module 4 (definition‑of‑done checker), and module 5 (orchestration decisions based on StepOutcome, FinalizeOutcome, RunResult).

Design Checklist Before Deploying the Observation Layer

Every tool result must be serialized uniformly; large payloads should only go to trace.

Partial failures must be written back item‑by‑item; successful items must not be dragged down.

An error‑grading table with L0 auto‑retry limits must exist; retryable: false must forbid silent retries.

Repeat‑loop detection (same tool + args, same error + args, critique spin) must be implemented.

Definition of Done (module 4) must be available for rule‑based or critique checks.

Closing critique should be enabled according to risk level; critique rounds must have a hard upper bound.

Degraded final output must clearly state completed parts, missing parts, and actionable suggestions.

Fact‑based answers must cite the originating tool or RAG source; on total failure the system must refuse rather than hallucinate.

Structured outputs need a validate‑plus‑repair path.

Trace must record error_code, retry count, critique verdict, and termination reason (module 11).

Mapping to Real‑World Projects

Observation serialization and write‑back correspond to practical modules 2‑3 (tool result → next Decide).

L0 auto‑retry and error write‑back are demonstrated in module 3 (think → act → observe).

Repeat‑loop detection and max‑steps degradation are required in module 3 (essential for production).

Done checking and closing critique appear in modules 4‑5 (multi‑city weather example).

Source citation and structured final output are covered in module 8.

Trace recording and evaluation are handled in module 11.

Summary

Observation is the quality‑check input for the next reasoning step; it must be short, structured, and clearly indicate success or failure, with partial failures written back individually.

Error handling requires grading (L0‑L4); retryable is a hard switch that determines whether the orchestration layer may silently retry.

Self‑critique occurs both intra‑step and at closing, enabled by risk level and limited to a few rounds to avoid infinite reflection.

Closing relies on a Definition of Done checklist and output constraints; no tool_calls does not mean completion, and degraded finals must honestly report progress.

Repeat‑loop detection acts as the brake for the orchestration layer, complementing max_steps and preventing endless loops.