From RAG to GraphRAG: How Huolala Raised Metadata Retrieval Accuracy from 56% to 78%

1. What is RAG?

RAG (Retrieval‑Augmented Generation) combines information retrieval with text generation. Before generating an answer, the system retrieves relevant evidence from a large knowledge base, improving accuracy, timeliness, and reducing hallucinations.

2. Challenges of Traditional RAG

RAG faces multiple issues: trade‑offs between relevance and freshness, vector recall missing key information, difficulty handling complex knowledge structures, and semantic gaps between user intent and retrieved content. Optimisation requires scenario‑specific tuning, better retrieval algorithms, and knowledge filtering.

3. Introducing GraphRAG

GraphRAG extends RAG by incorporating a Knowledge Graph. Instead of only vector‑based document retrieval, it builds a structured graph of entities and relationships, using graph algorithms (community detection, centrality) to retrieve sub‑graphs as context, which enhances reasoning over complex queries.

4. GraphRAG Architecture

Offline stage: Raw metadata is chunked, LLM extracts entities and relations, embeddings are stored in a Vector DB, and entities/relations are indexed in a Graph DB.

Online stage: User query is embedded, combined vector and graph retrieval fetches relevant text chunks and graph entities, which are merged into a prompt for the LLM to generate answers for recommendation, search, or dialogue scenarios.

5. Core Features of Graph‑based RAG

Multi‑index combination: graph, vector, full‑text indexes.

Hybrid retrieval: vector, BM25, scalar retrieval.

Multi‑hop reasoning based on graph relationships.

6. Comparison of Graph‑based RAG Variants

Three typical paradigms are GraphRAG (Microsoft open‑source), LightRAG, and PathRAG, each focusing on knowledge mining, lightweight deployment, or path reasoning. LightRAG was chosen for its balance of complexity, flexibility, and embed‑ability.

7. Naive RAG (Version 1.0) – Problems and Bad Cases

Overall accuracy was only 55% (≈60% Top‑K). Main issues:

Knowledge base “malnutrition” – only schema and comments, lacking business context and data lineage.

Single‑mode vector retrieval – poor recall for synonyms, multi‑entity queries.

No boundary awareness – the system cannot recognise out‑of‑scope questions.

Chunking strategies were evaluated. Whole‑table chunks preserved completeness but caused mismatched similarity and token overflow; field‑level chunks reduced these problems and were adopted.

1. 用户问题：xxx字段，在哪个表里能找到？
2. 向量数据库检索结果：{"tableName":"table1","column":"xxx","type":"string","comment":"xxx"}; ...
3. LLM生成答案：xxx字段在"table1"这个表中可以找到

Bad‑case analysis highlighted semantic mismatch, low multi‑entity recall, and irrelevant information interference.

8. GraphRAG (Version 2.0) – Design Details

Knowledge graph entities are classified into tables/fields, business terms, and synonym layers. Entity weighting formulas are defined, e.g.,

TableScore = manual_boost_1 * (w1 * score_downstream + w2 * score_popularity + w3 * score_star)
FieldScore = manual_boost_2 * (w4 * base_score + w5 * table_factor)

The indexing pipeline merges multi‑source metadata (tables, lineage, documentation) into Graph Storage and links to vector chunks. Retrieval flow: LLM extracts high‑level and low‑level keywords, expands low‑level terms via synonym dictionary, performs hybrid retrieval (vector + BM25), re‑ranks to obtain Top‑K entities, then combines local (text‑chunk) and global (graph‑based) contexts for LLM generation.

9. Results

Overall accuracy improved from 56% to 78%.

Knowledge recall 91%, Top‑K hit rate 90%, MRR 0.73.

AI‑based retrieval penetrates traditional keyword search by ~30% and saves >20% time in data‑warehouse Q&A.

10. Future Work

Upgrade hybrid retrieval and ranking (add full‑text and scalar search, refine entity weight models).

Enrich the knowledge base with automated term extraction and semantic expansion.

Explore Agentic RAG: planning‑capable agents, multi‑hop reasoning loops, and proactive clarification.

Conclusion

The shift from RAG to GraphRAG shows that metadata retrieval hinges on organizing metadata as a graph; pure semantic similarity is insufficient. By jointly recalling entities and relations, recall and accuracy rise dramatically, and the bottleneck moves from the large model to the retrieval and knowledge organization layers.