Why Vector Databases Fail Certain RAG Tasks and How Knowledge Graphs Solve Them

Choosing a retrieval method for a RAG system is not a binary decision; it depends on the shape of your data and how users ask questions. The author recounts three projects—pure vector retrieval, pure knowledge‑graph retrieval, and a hybrid approach—and stresses that the wrong choice makes later tuning almost impossible, while the right choice often works out‑of‑the‑box.

When Vector Databases Are Sufficient

If the knowledge base consists mainly of unstructured text and queries are vague or conversational (e.g., “What is the travel‑expense reimbursement process?”), a vector database alone can handle the task. The core capability is semantic similarity: documents are split into chunks, embedded, and the query embedding is used to find the nearest neighbours. The typical pipeline looks like:

from langchain.vectorstores import Milvus
from langchain.embeddings import OpenAIEmbeddings
# 1. Split & embed documents
chunks = text_splitter.split_documents(documents)
vectorstore = Milvus.from_documents(chunks, OpenAIEmbeddings())
# 2. Retrieve top‑k similar chunks
query = "出差报销的流程是什么"
relevant_chunks = vectorstore.similarity_search(query, k=5)
# 3. Generate answer with LLM
answer = llm.generate(context=relevant_chunks, question=query)

Advantages include friendliness to unstructured text, strong semantic understanding, and fast engineering rollout with mature open‑source vector stores such as Milvus, Pinecone, Weaviate, and Qdrant.

Limitations of Pure Vector Retrieval

Vector search struggles with relational or multi‑step questions. For example, answering “Who is responsible for Project A and what other projects do they manage?” requires two reasoning steps: first find the owner of Project A, then find the other projects that owner handles. Vector retrieval only returns chunks that mention Project A, missing cross‑document relationships.

Mitigations tried by the author include increasing top‑k (5 → 20), using HyDE (hypothetical document generation), and multi‑query strategies, but these are merely patches that do not solve the fundamental inability to perform multi‑hop reasoning.

Another pitfall is exact matching of numbers or dates; a query like “2024 Q3 sales” may retrieve 2023 or Q2 content because vector similarity is fuzzy.

Strengths of Knowledge Graphs

Knowledge graphs represent facts as triples (entity‑relationship‑entity) and enable multi‑hop graph queries. For the same ownership question, a Cypher query can retrieve precise results in two hops:

// Cypher two‑hop query
MATCH (p:Project {name: "A项目"})<-[:负责]-(person)-[:负责]->(other:Project)
WHERE other.name <> "A项目"
RETURN person.name, collect(other.name) AS other_projects

Benefits include exact reasoning, clear explainability (the path is visible), and easy updates—changing a relationship only requires editing a single edge. These properties are essential in compliance‑heavy domains such as finance, healthcare, and law.

Challenges of Knowledge Graphs

Building a graph demands a costly knowledge‑extraction pipeline: named‑entity recognition, relation extraction, and entity disambiguation. The author reports ~85 % accuracy for simple relations (e.g., “张三 belongs to 技术部”) but <60 % for complex ones (e.g., “系统 X depends on service Y’s callback”). Schema design is also critical; a poorly defined schema can make later extensions painful, as illustrated by mixing “temporary” and “formal” responsibility edges.

Hybrid Approaches

In practice, most projects converge on a hybrid solution. Three patterns are described:

Scheme 1 – Vector‑first, Knowledge‑Graph supplement

# Hybrid retrieve (pseudo‑code)
def hybrid_retrieve(query):
    # 1. Vector retrieval
    vector_results = vectorstore.similarity_search(query, k=5)
    # 2. Entity extraction from query
    entities = extract_entities(query)  # e.g., ["A项目"]
    # 3. Graph lookup for each entity
    graph_results = []
    for entity in entities:
        related = knowledge_graph.query(
            f"MATCH (n)-[r]->(m) WHERE n.name='{entity}' RETURN n,r,m"
        )
        graph_results.extend(related)
    # 4. Merge and feed to LLM
    context = merge(vector_results, graph_results)
    return llm.generate(context=context, question=query)

This works well when most queries are about “what” or “how”, while relational questions are answered via the graph.

Scheme 2 – Knowledge‑Graph first, Vector fallback

# Graph‑first retrieve (pseudo‑code)
def graph_first_retrieve(query):
    cypher = text_to_cypher(query, schema=graph_schema, few_shots=examples)
    if not validate_cypher(cypher):
        return fallback_to_vector(query)
    graph_results = knowledge_graph.execute(cypher)
    if not graph_results:
        return fallback_to_vector(query)
    return llm.generate(context=graph_results, question=query)

The bottleneck is reliable Text‑to‑Cypher conversion; the author supplies full schema and few‑shot examples, validates syntax, and falls back to vector search when the generated query is empty or unreasonable.

Scheme 3 – Graph RAG (Microsoft GraphRAG)

GraphRAG builds a community‑level graph from documents using LLMs and performs multi‑layer retrieval. It excels at global summarisation (e.g., “What is the overall architecture of this codebase?”) but incurs huge token costs and performs poorly on fine‑grained fact queries.

Selection Guide

The author provides a decision table (image) summarising when to use each approach. In short:

Simple, unstructured text + semantic queries → vector DB.

Complex relationships, multi‑hop, precise matching → knowledge graph.

Mixed needs → start with vector DB for quick rollout, then augment critical relational scenarios with a graph.

Common Pitfalls

Trying to force multi‑hop reasoning on pure vector retrieval leads to exponential noise; use a graph instead.

Knowledge‑graph cold‑start can take months due to schema design and triple extraction.

Embedding model choice matters; domain‑fine‑tuned models outperform generic sentence‑transformers.

Over‑engineering the routing layer; simple rule‑based routing (e.g., presence of explicit entity names) is often more reliable than complex classifiers.

Conclusion

The choice between vector databases and knowledge graphs ultimately hinges on data complexity and query patterns. Vector retrieval is optimal for semantic, unstructured queries; knowledge graphs excel at relational, multi‑hop, and compliance‑driven queries. Most real‑world projects benefit from a hybrid architecture that starts with a vector store for rapid deployment and adds a graph layer for the relational edge cases.