Challenges and Practices of LLM‑Based Knowledge Bases and Personal Assistants

The article discusses the typical workflow of large language models (LLMs) – feeding a prompt and receiving an answer – and highlights two fundamental problems: managing historical dialogue context and the limited token window.

Two application scenarios are examined: knowledge‑base question answering and personal‑assistant agents. For each, the author evaluates practical difficulties and possible remedies.

Open‑source agent frameworks such as LangChain , AutoGen , and MetaGPT are introduced. LangChain focuses on a single agent’s service integration, AutoGen enables multi‑agent collaboration, and MetaGPT claims to generate product documentation, test code, and runnable code from high‑level requirements.

A concrete prompt template used by LangChain is shown:

PREFIX = """Answer the following questions as best you can. You have access to the following tools:"""</code><code>FORMAT_INSTRUCTIONS = """Use the following format:</code><code>Question: the input question you must answer</code><code>Thought: you should always think about what to do</code><code>Action: the action to take, should be one of [{tool_names}]</code><code>Action Input: the input to the action</code><code>Observation: the result of the action</code><code>... (this Thought/Action/Action Input/Observation can repeat N times)</code><code>Thought: I now know the final answer</code><code>Final Answer: the final answer to the original input question"""</code><code>SUFFIX = """Begin!"""</code><code>Question: {input}</code><code>Thought:{agent_scratchpad}"""

In the knowledge‑base setting, documents (PDF, Word, etc.) are split, embedded, and stored in a vector database. Retrieval then supplies relevant chunks to the LLM. The main challenges are:

Multimodal content (images, video) that cannot be represented by plain text embeddings.

Large document collections that produce too many retrieved chunks, exceeding token limits.

Typical components to address these issues are a Text Splitter , an Embedding model (e.g., OpenAI’s text‑embedding‑ada‑002), and a Vector Store (e.g., Holo, Redis).

Personal‑assistant agents follow a ReAct‑style loop: the LLM decides whether to invoke a tool, constructs tool arguments, the tool is called, and the LLM processes the result. Challenges include:

Tool parameters must be simple; complex structures often cause parsing errors.

Each call carries the full conversation history, leading to token overflow for long contexts.

The author argues that fine‑tuning (including LoRA‑based methods for text‑to‑image models) is essential to overcome these limitations, especially for domain‑specific or multimodal knowledge.

Future expectations include LLM inference reaching ChatGPT‑4 quality, making fine‑tuning a standard step for specialized applications, and better integration of streaming capabilities (e.g., in DingTalk).