Building a Judgmental RAG Agent: From Simple Search to Autonomous Search‑Filter‑Dive‑Validate Loop

Overview

The core challenge of knowledge Q&A is achieving both high recall and precision while handling multimodal inputs and permission isolation; a powerful Agent runtime is required to meet these demands.

AgentScope 2.0 HarnessAgent Architecture

We adopt AgentScope 2.0 because its HarnessAgent architecture provides the engineering capabilities needed for long‑running agents in production.

Key Capabilities

ReAct Loop : The agent iterates through reason‑act‑observe cycles, enabling a "search → evaluate → deep‑read → re‑search" workflow.

Middleware : Custom logic can be injected at critical points without modifying core code; we use the onActing hook to apply a three‑stage quality filter transparently.

Parallel Tool Calls : When the agent decides to invoke multiple tools, the framework executes them in parallel, allowing "multiple tools + multiple variants" retrieval.

Composite Retrieval Process

The agent performs multi‑source parallel retrieval (knowledge base, Feishu docs, Feishu messages, Feishu notes), generates diverse query variants, and evaluates results for relevance, completeness, and timeliness. If the evaluation deems the information insufficient, the agent decides whether to deep‑read the full document, perform supplemental searches, or cross‑validate conflicting sources.

Autonomous Decision Making

Each round follows a "search → evaluate → decide" loop, with the agent autonomously choosing which tools to call, how many searches to run, whether to deep‑read, and when to stop.

Query Expansion

Before retrieval, the original user question is expanded into full natural‑language sentences rather than keyword bags, because embedding models are trained on complete sentences and produce more accurate semantic vectors.

Three‑Stage Quality Pipeline

Retrieved results pass through three stages:

FastPass : If ≤2 results have original scores ≥0.7, they are returned directly with zero latency.

Reranker : A cross‑encoder (gte‑rerank‑v2) re‑scores snippets; items with scores < 0.3 are filtered, and the top‑8 are kept.

LLM Grading : An LLM grades each item (0.1–1.0); items with scores < 0.5 are discarded, and the remainder are sorted by relevance.

// Implement MiddlewareBase to inject filtering logic before/after tool execution
Flux<AgentEvent> onActing(Agent agent, ActingInput input, next) {
    if (!config.isEnabled()) return next.apply(input); // pass‑through if disabled
    // 1. Identify if this round includes a knowledge_search tool call
    targetToolCalls = input.toolCalls().filter(tc -> tc.name == "knowledge_search");
    if (targetToolCalls.isEmpty()) return next.apply(input); // no target tool, pass‑through
    // 2. Let the tool execute, then collect all events for unified processing
    events = next.apply(input).collectList();
    return processEvents(events, targetToolCalls); // enter three‑stage filter
}

List<RetrievedItem> filter(String query, List<RetrievedItem> items) {
    // ── Stage 0: FastPass ──
    if (items.size() <= 2 && items.allMatch(i -> i.originalScore >= 0.7)) {
        return items; // high confidence, skip further filtering
    }
    // ── Stage 1: Reranker coarse filter ──
    if (config.rerankerEnabled) {
        scores = rerankerService.rerank(query, items.snippets); // gte‑rerank‑v2
        items = items.filter(i -> scores[i.index] >= 0.3)
                     .sortByDescending(i -> i.rerankerScore)
                     .limit(8);
    }
    // ── Stage 2: LLM Grading fine filter ──
    if (config.llmGradeEnabled && !items.isEmpty()) {
        graded = llmScoringService.gradeInBatch(query, items); // max 10 items, 30s timeout
        items = graded.filter(g -> g.score >= 0.5)
                     .sortByDescending(i -> i.relevanceScore);
    }
    return items; // filtered results replace original tool output transparently
}

Multimodal Support

Users can upload screenshots; the system detects image context and automatically switches to a vision‑language model. The model selection logic ensures that pure‑text models reject image inputs, preventing failures. Historical images are also considered in each round because the LLM is stateless.

Image Lifecycle Management

Upload: drag‑drop or paste, up to 6 images, type/size validation.

Send: read bytes from internal object storage, Base64‑encode, and embed in multimodal messages.

History Replay: images are replayed per message, filtered by user scope, with total size budgeted.

Answer Embedding: agents can insert retrieved images into answers with unified numbering.

Production‑Grade Reliability

SSE (Server‑Sent Events) breakpoint‑resume is required not only for server crashes but also for normal user actions such as page backgrounding, refresh, tab reopening, or network changes.

Three Recovery Paths

Path 1 – Same‑Instance Resume : The browser reconnects to the same instance; in‑memory agent state is used to send a snapshot and continue streaming.

Path 2 – Cross‑Instance Forwarding : Load balancer routes the reconnect to a different instance; the new instance looks up the original instance via a Redis routing table and forwards the SSE stream.

Path 3 – Snapshot Reconstruction : If the original instance is unreachable, the system rebuilds the full session from Redis snapshots, ensuring no data loss.

Design Decisions

We avoid Pub/Sub for event delivery because of subscription latency; Redis stores only routing information. Heartbeat messages every 10 seconds keep the SSE connection alive. A Redis SETNX distributed lock prevents duplicate concurrent requests from double‑clicks.

Model Failover

When the primary LLM returns 5xx, rate‑limit, timeout, or network errors, the system automatically switches to a standby model; 4xx client errors are returned directly for user correction.

Key Innovations and Outlook

Compared with standard RAG “search‑then‑concatenate” approaches, our system differs in six ways:

Composite retrieval is driven by agent‑level decision making, not just multi‑source crawling.

The quality pipeline includes three distinct stages (FastPass, Reranker, LLM Grading) rather than a single reranker.

SSE breakpoint‑resume uses routing‑table + internal forwarding instead of Pub/Sub.

Future work includes short‑term fine‑tuned summarisation per data source (Feishu docs, messages, notes) and long‑term vision of a personal knowledge assistant that combines public knowledge, personal data, and persistent memory.