Why Direct Prompting Beats LLM Knowledge‑Base Agents for Test‑Case Generation

Question

Scenario 1: The user types the software registration requirements directly into the LLM prompt and asks for test cases. Scenario 2: The user first stores the same requirements in a knowledge‑base agent, then asks the agent to generate test cases based on that stored knowledge. The output from Scenario 1 is noticeably superior.

Agent Answer

The difference stems from the core mechanism of large language models: the contrast between direct context learning and knowledge‑base retrieval.

1. Attention Mechanism – Input Window vs. Retrieval

Scenario 1 places the requirements inside the model’s input window, allowing the attention mechanism to process the text in full, preserving order and detail.

Scenario 2 adds a retrieval step: the query is vectorised, the most similar fragments are fetched, and then the model generates an answer. This introduces two risks:

Retrieval truncation – only a few relevant passages are returned, potentially omitting important details.

Context fragmentation – the retrieved snippets are pieced together, breaking the coherent logical flow of the original document.

2. LLM’s “Instant Memory” Is Strongest When Information Is Immediate

In Scenario 1 the requirements are right in the conversation context, giving the model its highest processing precision.

In Scenario 2 the requirements reside in a vector database; each query forces the model to “think while it searches,” and every retrieval step compresses the information, degrading output quality because current retrieval technology cannot guarantee loss‑free extraction of all logical relations.

3. Retrieval Is Subject to “Garbage‑In, Garbage‑Out”

The agent relies on an embedding model to convert the requirements into vectors and then matches by semantic similarity, which does not equal logical relevance. For example, a requirement states: “If the username contains special characters, registration fails with ‘format error’.” When asked to design test cases, the retrieval may surface the phrases “registration fails” and “format error” but miss the testing techniques such as equivalence partitioning or boundary‑value analysis, or may fail to capture the exact definition of “special characters.”

Conclusion and Recommendations

For single, medium‑length requirements, place the full text directly in the prompt; this yields the most accurate results as long as the input length stays within the model’s context window.

Knowledge‑base agents become valuable only when dealing with very long documents (tens of thousands of words) or when the task spans many separate documents, situations where the prompt cannot contain all the necessary information.

Thus, the phenomenon is not a flaw in agent technology but a reminder that for isolated, moderately sized specifications, direct prompting remains the most efficient and faithful approach.