Building a Vertical Domain QA Bot with Vector Search, RAG, and SFT

With the rapid rise of large language models (LLMs), the barrier to building AI applications has dropped dramatically. Starting at the end of last year we experimented with LLMs and launched a domain‑specific answer‑bot for logistics. This article records the learning‑by‑doing process for beginners who want to create small AI tools or improve daily workflows.

Target audience : entry‑level users interested in LLMs, looking for low‑effort ways to build utilities or boost operational efficiency.

Background : Traditional QA bots rely on hierarchical menus or keyword matching, which cannot quickly and accurately retrieve the desired answer. Our goal is to build a bot for internal staff that answers precisely, responds within seconds, and refuses out‑of‑scope queries.

Stage 1 – Vector Search : Embed texts (questions, answers, and documents) into high‑dimensional vectors, store them in a vector database, and perform semantic similarity search. This approach mimics relational DB lookup but matches on meaning rather than exact terms. The workflow includes data preparation (cleaning knowledge‑base into QA pairs) and online inference (convert user query to a vector, retrieve the nearest record, and apply a similarity threshold).

Stage 2 – Retrieval‑Augmented Generation (RAG) : Combine retrieval with generation. First retrieve N relevant passages from the vector store, then let an LLM generate a detailed answer based on those passages. This improves accuracy for domain‑specific questions while still leveraging the LLM’s language capabilities.

Stage 3 – Supervised Fine‑Tuning (SFT) : Fine‑tune a smaller base model (e.g., 7B) on high‑quality, manually labeled QA data. Techniques such as LoRA, full‑parameter fine‑tuning, and instruction tuning are discussed. Training hyper‑parameters (epochs, learning rate, batch size) are adjusted to avoid over‑fitting and achieve a loss below 0.1.

Stage 4 – Hybrid Strategies : Combine SFT with vector search (reject answers when the fine‑tuned model’s output diverges from retrieved results) and optionally fall back to RAG for out‑of‑domain queries.

Stage 5 – Engineering Optimizations : Add user‑experience features (thumbs up/down, multimedia answers, similar‑question list), automate data cleaning and model deployment pipelines, and continuously enrich the knowledge base from frontline feedback.

Conclusion : AI‑driven QA systems require a blend of retrieval, generation, and fine‑tuning to meet strict accuracy and latency requirements. Continuous learning, cost‑aware deployment, and robust evaluation are essential for sustainable engineering practice.