When Should an AI Agent Query the Knowledge Base? Intent Recognition & Query Rewriting
The article explains how AI agents should autonomously decide whether to query a knowledge base by recognizing user intent, rewriting ambiguous queries with context, and filtering retrieved information to avoid hallucination, using examples of rewriting notices versus answering policy questions.
First Look at What the User Wants to Do
The previous article explained that RAG solves the problem of models not knowing and not being allowed to hallucinate: retrieve documents first, then feed them to the model. However, retrieval is triggered programmatically every time before answering.
In real business scenarios, an Agent must decide autonomously: does this question actually need a lookup?
Assume a company deploys an AI office assistant. An employee asks:
Rewrite the company notice "The reimbursement system will be under maintenance tonight at 8 PM, please submit expenses in advance" to make it more formal and official.
The intent is simply to rephrase the notice, but the assistant queries the reimbursement policy and appends "Please complete manager approval first." The maintenance notice turns into a reimbursement requirement, leaving employees confused.
The same keyword "reimbursement" appears in three different user intents:
"Make this reimbursement notice formal" – content is already provided; only wording adjustment is needed.
"What materials are required for business trip reimbursement?" – answer lies in company policy; the model cannot rely on common sense and should query.
"Where is my reimbursement approval at?" – requires real-time record from the business system; if the knowledge base only contains policy files, querying it won't help; a tool that checks approval status is needed.
Therefore, don't rush to match keywords. First understand what the user wants to do, then check if the information at hand is sufficient. Need company policy? Query the policy. Only processing user-provided text? Usually no query needed. Explaining a common term may not require internal documents either.
This is the basic intent recognition approach for an Agent: treat retrieval as a tool, let the model decide whether to issue a call request, and have the program execute it. Not every question must trigger a knowledge base lookup.
During implementation, the tool description must clearly state what the library contains and what problems it solves. For example:
Tool description example:
Query internal company policies, including reimbursement, leave, procurement documents.
Use when user asks about company regulations.
Does not provide real-time approval status of personal expense items.Only then does the LLM have a basis for tool selection. Correctness isn't guaranteed, so the program should log invocation records for later inspection of what was searched and retrieved.
Before Querying, Complete the Question
An employee follows up:
Conversation example:
Previous Q: What materials are needed for domestic business trip accommodation reimbursement?
This Q: What about electronic invoices?
Query passed to retrieval tool:
Company domestic travel reimbursement policy accommodation electronic invoice submission requirementsHumans understand "What about electronic invoices?" in context, but a search tool may not. Using only the fragment could return procurement invoice rules or invoicing operation guides. Including the previously established context "domestic travel" and "accommodation" makes it easier to find the needed policy.
This is called query rewriting : keep the meaning, turn chat language into search language, but do not rewrite into "Is electronic invoice required to be printed and signed?" The user never mentioned printing or signing; the model must not guess a requirement and then search around that guess. Similarly, if the user didn't specify which subsidiary, the model must not pick one. When missing information affects the answer, ask the user to clarify.
Retrieved Documents Should Not Be Used Blindly
After retrieval, there are two ways to handle the information:
Direct injection: Put the retrieved documents into the model input for this turn, letting the model read and answer. No separate selection step; the model judges relevance during answering.
Relevance filtering first: Let the model judge "Can this document answer the current question?" Keep suitable segments, then compose the answer. For example, querying accommodation electronic invoices returns supplier procurement invoice requirements. Both mention "invoice" but the business context differs; they cannot be mixed. Also check sufficiency: if the file only says "electronic invoices supported" without stating whether printing is required, you cannot conclude "no printing needed."
State what is certain, note what is missing. If necessary, reformulate the query and search again; if still no evidence, do not fabricate.
Pre-filtering can also misjudge; an extra model call doesn't guarantee correctness. Not every question should query the knowledge base; querying incorrectly can be worse than not querying at all. Extra documents can bias the model, causing it to include unasked information in the answer.
How Did a Maintenance Notice Gain an Approval Requirement?
Return to the opening scenario. Suppose the assistant returned:
The reimbursement system will undergo maintenance at 20:00 tonight. Please submit expenses in advance and complete manager approval.
The original notice had no approval requirement. Where did it come from?
Checking invocation logs shows the model saw "reimbursement" and "submit expenses", so it searched the reimbursement policy. The retrieved snippet mentioned manager approval, which it then added to the notice.
Two errors occurred: first, treating "rewrite notice format" as "consult policy"; second, treating the retrieved policy as the arrangement for this maintenance. Even if the policy content is true, it doesn't prove approval must be completed before maintenance.
To fix, the instruction should state: When the user has provided all content and only requests rewriting, process directly. Preserve original intent, do not add new business requirements.
After correction, the same input should not trigger a retrieval request and should return only:
The reimbursement system will undergo maintenance at 20:00 tonight. Please submit expenses in advance.
If the employee then asks "What approvals are needed before submission?", that is when the company policy should be queried.
An Agent 's job is to perform intent recognition based on user input, deciding whether to query, what to search for, and which materials can answer the question.
With tools, RAG, and loops in place, should you use an Agent or a fixed Workflow (predefined step execution) for better reliability? Not every scenario warrants an Agent — the next article will cover selection criteria.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
