Building an Enterprise‑Grade RAG Knowledge Base from Scratch: Architecture, Tech Choices & Pitfalls

Introduction

Insurance agents need fast, accurate answers to questions like “What is the surcharge standard for Type‑2 diabetes?” Traditional keyword search often fails, and using a generic large model can hallucinate non‑existent policy clauses. Retrieval‑Augmented Generation (RAG) solves this by first retrieving relevant document fragments and then generating evidence‑backed answers.

System Overview

The author built an enterprise‑grade RAG platform called YunZhi for the insurance domain. Core capabilities include knowledge‑base management with three visibility levels (PRIVATE, TEAM, PUBLIC), support for 100+ document formats, streaming RAG answers with source citations, multi‑turn dialogue, hybrid search (vector + keyword), fine‑grained RBAC/ACL permissions, and audit tracing for low‑confidence results.

Architecture Design – Why Java + Python?

The system uses a Vue 3 SPA front‑end, Nginx reverse proxy, a Java Spring Boot service (handling authentication, CRUD, permission checks, and vector search via pgvector) and a Python FastAPI service (document parsing with Unstructured , embedding via BGE‑M3, ChromaDB vector store, and BGE‑Reranker). The split lets each language play to its strengths: Java for robust enterprise features (JWT, RBAC, transactional APIs) and Python for AI‑heavy tasks (NLP, transformers). Communication between services is via REST API with API‑Key authentication.

Spring AI 2.0 Role

Spring AI is used on the Java side for two purposes: calling the LLM through an OpenAI‑compatible interface that connects to Xiaomi’s MiMo model, and performing vector search directly with the built‑in pgvector integration.

Technology Selections

LLM Choice – MiMo vs. GPT‑4

Cost: MiMo (very low) vs. GPT‑4 (high)

Chinese capability: both excellent

Compliance: MiMo is a domestic model, keeping data in‑country; GPT‑4 involves data outbound risk.

Interface compatibility: MiMo offers OpenAI‑compatible endpoints, making integration cheap.

Data compliance is the decisive factor for insurance documents.

Embedding Model – BGE‑M3

Multilingual support, strong performance on mixed Chinese‑English content.

1024‑dimensional vectors balance accuracy and speed.

Open‑source and free, allowing local deployment without API costs.

Pairs well with BGE‑Reranker‑v2‑m3 for consistent scoring.

Embedding runs on the Python side via HuggingFace Sentence‑Transformers; the Java side uses bge-large-zh-v1.5 with pgvector.

Object Storage – RustFS over MinIO

MinIO changed its license to AGPL v3 in 2021, restricting commercial use.

RustFS is an Apache‑2.0‑licensed S3‑compatible storage rewritten in Rust, offering better performance.

API compatibility means only the endpoint and credentials need updating.

Front‑end Stack

Vue 3.5 + TypeScript + Vite 6

Ant Design Vue 4 for enterprise UI components

Pinia for state management

SSE streaming rendering that bypasses Vue’s reactivity for performance.

RAG Pipeline – From Question to Answer

Step 1: Query Rewrite – The user’s informal query “糖尿病加费” is rewritten to “2型糖尿病核保加费标准及费率表” by a Java QueryRewriteService that calls the LLM for intent detection and keyword expansion.

Step 2: Hybrid Search – The rewritten query follows two paths:

Semantic search (vector similarity) using BGE‑M3 embeddings and cosine similarity in ChromaDB, capturing semantic matches like “加费” ↔ “费率上浮”.

Keyword search (full‑text BM25) via PostgreSQL tsvector, ensuring exact matches for domain‑specific terms such as “甲状腺结节”.

Results are merged with Reciprocal Rank Fusion (RRF) to balance semantic understanding and precise matching.

Step 3: Reranker – The top 15 candidates are re‑scored by BGE‑Reranker‑v2‑m3 (a cross‑encoder) and the best 5 are kept, providing finer granularity than the initial Bi‑Encoder similarity.

Step 4: Context Building + LLM Generation – The selected fragments, system prompt, and conversation history are sent to the MiMo LLM. The model generates an answer, which the Java side streams back via Spring AI chatModel.stream() and Server‑Sent Events.

Step 5: Audit Marking – If the highest similarity score is below 0.5, the system creates a PENDING audit log, prompting administrators to verify knowledge‑base coverage.

Streaming Rendering Performance Optimisation

Because SSE can push dozens of chunks per second, updating Vue’s reactive state for each chunk caused severe UI lag. The solution mirrors approaches used by ChatGPT and Claude:

During streaming, only plain text is displayed; Markdown rendering is deferred until the stream ends.

Content is accumulated in a local variable and written directly to the DOM with textContent inside requestAnimationFrame, bypassing Vue’s virtual‑DOM diff.

Code example:

let streamingContent = ''
let rafPending = false
function flushStreaming() {
  rafPending = false
  const el = container.querySelector('[data-msg-id="' + msgIndex + '"] .msg-text')
  if (el) el.textContent = streamingContent // direct DOM update
}
// each SSE chunk
streamingContent += payload
if (!rafPending) {
  rafPending = true
  requestAnimationFrame(flushStreaming) // RAF throttling, one update per frame
}
// after stream ends
assistantMsg.content = streamingContent
assistantMsg.isStreaming = false
assistantMsg.renderedContent = renderMarkdown(streamingContent)

This approach eliminates repeated re‑renders, keeping the UI smooth even under high throughput.

Permission System – RBAC + ACL Hybrid Model

Insurance knowledge requires strict access control. The system defines three visibility scopes (PRIVATE, TEAM, PUBLIC) and four role levels (OWNER > ADMIN > EDITOR > VIEWER). RAG queries automatically check the user’s SEARCH permission, ensuring that only authorized knowledge bases are queried.

Pitfalls and Resolutions

SSE buffering in Nginx – Disabled proxy_buffering, proxy_cache, and extended proxy_read_timeout for the streaming endpoint.

BGE‑M3 similarity threshold – Lowered the cosine threshold from 0.6 to 0.3 to increase recall before reranking.

Vue reactivity causing UI freeze – Implemented the direct‑DOM streaming technique described above.

Spring AI multiple starter conflict – Resolved by using @Qualifier or enabling only one ChatModel provider in application.yml.

RustFS health‑check failure – Adjusted the Docker Compose health‑check script to expect HTTP 403 from the root path.

Project Metrics

Code files: 176 | Symbol nodes: 2,762 | Dependency edges: 5,292 | Feature clusters: 97 | Database migration versions: V1 → V10 | Docker services: 8.

Conclusion

The project demonstrates a pragmatic approach: use Java for business logic, Python for AI, choose a compliant low‑cost domestic LLM (MiMo), bypass framework bottlenecks with direct DOM manipulation, and combine hybrid search with a reranker to achieve reliable enterprise knowledge‑base retrieval. The open‑source stack (Spring Boot 4.0.7, Spring AI 2.0, FastAPI, Vue 3.5, PostgreSQL 16, ChromaDB, BGE‑M3) is available for anyone building similar systems.