What Is Hybrid Search in RAG and Why Choose It Over Pure Vector Retrieval?

Interview Focus Points

Concept depth : Interviewer wants to see if you understand the strengths and limits of dense (semantic) and sparse (keyword) retrieval.

Engineering practice : Have you actually deployed hybrid retrieval in production? How to configure RRF, weight tuning, and selection criteria?

Architecture design : Hybrid retrieval is not just concatenating two result lists; it requires proper fusion, re‑ranking, and evaluation.

Core Answer

Hybrid Search = Dense (Semantic) Retrieval + Sparse (Keyword) Retrieval, followed by result fusion and ranking.

Why combine both? Each method has blind spots:

Dense Retrieval excels at semantic similarity, synonyms, and abstract concepts, but struggles with exact keyword matches such as product model numbers ( iPhone 15 Pro Max), error codes ( ERR-4023), or function names ( findByIdAndName).

Sparse Retrieval (BM25) excels at exact matching of proper nouns, code identifiers, and numbers, but cannot understand paraphrased intent (e.g., “how to unsubscribe” vs. “cancel subscription”).

Hybrid Retrieval combines the strengths of both, at the cost of slightly higher implementation complexity.

One‑sentence summary: Dense retrieval grasps “meaning”, keyword retrieval grabs the “literal”, and hybrid retrieval fuses them to cover each other's gaps.

Overall Architecture

The workflow consists of three steps:

Parallel retrieval: the query is sent simultaneously to dense and sparse channels, each returning Top‑K results (typically K=10‑20).

Result fusion: RRF (Reciprocal Rank Fusion) or weighted scoring merges the two lists into a unified ranking.

Optional re‑ranking: a cross‑encoder model further refines the fused results.

Why Pure Vector Retrieval Is Insufficient

Vector retrieval encodes text into high‑dimensional embeddings and measures cosine similarity or inner product. This works well for “understanding meaning” but has a fatal weakness: weak exact‑keyword matching.

Spring AI integrates Milvus vector DB; the query spring-ai-milvus-store asks for the version number. Vector retrieval may return many documents about “Spring AI” and “Milvus” but rank the exact artifactId lower because the token is diluted in the embedding space.

BM25, on the other hand, sees the exact token spring-ai-milvus-store and can hit the target document directly.

Conversely, a query like Spring AI 怎么连接向量数据库？ may not contain the word “连接”. BM25 might miss it, while dense retrieval can infer that “连接” and “整合” are semantically equivalent.

This complementarity is why hybrid retrieval is needed.

RRF Fusion Algorithm – The Core Magic

The key step is merging the two result lists. Directly adding dense scores (0‑1) and BM25 scores (0‑100) is meaningless because the scales differ.

RRF avoids score normalization by looking only at rank positions:

RRF_score(d) = Σ 1 / (k + rank_i(d))

rank_i(d)

: rank of document d in the i‑th retrieval list. k: smoothing constant, usually 60.

Example: Document A is rank 1 in dense and rank 5 in BM25. RRF_score(A) = 1/(60+1) + 1/(60+5) ≈ 0.031 Document B is rank 3 in dense and rank 1 in BM25. RRF_score(B) = 1/(60+3) + 1/(60+1) ≈ 0.031 RRF’s philosophy: if a document ranks high in either list, it receives a high final score, even if its score in the other list is low. The algorithm was introduced by Cormack, Clarke & Büttcher (2009) and is now the default fusion strategy in Elasticsearch, Azure AI Search, OpenSearch, etc.

Code Example: LangChain4j + Elasticsearch Hybrid Search

Using the LangChain4j framework, Elasticsearch provides the most mature hybrid‑search integration.

Maven dependencies :

<dependencies>
  <!-- LangChain4j core -->
  <dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j</artifactId>
    <version>1.0.0-beta2</version>
  </dependency>

  <!-- LangChain4j Elasticsearch integration (hybrid) -->
  <dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-elasticsearch</artifactId>
    <version>1.0.0-beta2</version>
  </dependency>

  <!-- LangChain4j Ollama (local embedding model) -->
  <dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-ollama</artifactId>
    <version>1.0.0-beta2</version>
  </dependency>
</dependencies>

Java code (key steps highlighted):

import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.embedding.EmbeddingModel;
import dev.langchain4j.model.ollama.OllamaEmbeddingModel;
import dev.langchain4j.rag.content.Content;
import dev.langchain4j.rag.content.retriever.ContentRetriever;
import dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever;
import dev.langchain4j.store.embedding.elasticsearch.ElasticsearchEmbeddingStore;
import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import java.util.List;

public class HybridSearchDemo {
    public static void main(String[] args) {
        // 1. Configure Elasticsearch client
        RestClientBuilder restClientBuilder = RestClient.builder(
                new HttpHost("localhost", 9200, "http")
        );
        // 2. Configure embedding model (local Ollama model)
        EmbeddingModel embeddingModel = OllamaEmbeddingModel.builder()
                .baseUrl("http://localhost:11434")
                .modelName("nomic-embed-text")
                .build();
        // 3. Create Elasticsearch vector store
        ElasticsearchEmbeddingStore embeddingStore = ElasticsearchEmbeddingStore.builder()
                .restClientBuilder(restClientBuilder)
                .embeddingModel(embeddingModel)
                .indexName("knowledge_base")
                .build();
        // 4. Pure vector retrieval (baseline)
        ContentRetriever vectorRetriever = EmbeddingStoreContentRetriever.builder()
                .embeddingStore(embeddingStore)
                .embeddingModel(embeddingModel)
                .maxResults(5)
                .build();
        // 5. Hybrid retrieval (BM25 + kNN + RRF)
        ContentRetriever hybridRetriever = EmbeddingStoreContentRetriever.builder()
                .embeddingStore(embeddingStore)
                .embeddingModel(embeddingModel)
                .maxResults(5)
                .searchType(ElasticsearchEmbeddingStore.SearchType.hybrid())
                .build();
        // 6. Execute comparison
        String query = "怎么在 Spring AI 中整合 Milvus 向量数据库？";
        System.out.println("=== Pure Vector Results ===");
        List<Content> vectorResults = vectorRetriever.retrieve(query);
        vectorResults.forEach(c -> System.out.println(c.textSegment().text()));
        System.out.println("
=== Hybrid Results ===");
        List<Content> hybridResults = hybridRetriever.retrieve(query);
        hybridResults.forEach(c -> System.out.println(c.textSegment().text()));
    }
}

The crucial difference is the call to .searchType(ElasticsearchEmbeddingStore.SearchType.hybrid()), which triggers Elasticsearch to run both BM25 and kNN and fuse the results with RRF.

Hybrid Support in Major Vector Databases

Elasticsearch : native hybrid support, sparse method BM25, fusion strategy RRF (built‑in).

Milvus 2.5+ : native hybrid, sparse methods BM25 / SPLADE, fusion via weighted merging.

Weaviate : native hybrid, sparse BM25, weighted fusion.

Qdrant : native hybrid, sparse vectors, fusion via RRF or weighted.

OpenSearch : native hybrid, BM25, RRF built‑in.

Chroma : limited hybrid support, requires external sparse component, manual fusion.

If you use Milvus 2.5, you can store dense and sparse vectors in the same collection and issue a HybridSearchRequest. Spring AI currently lacks a native abstraction for hybrid search, but community discussions are ongoing.

Production Best Practices

Hybrid + Rerank is the golden combo : Azure AI Search experiments show pure BM25 as baseline, pure vector +8 % recall, hybrid +15 %, and hybrid + Rerank +25 %.

Weight tuning : Start with 0.5 : 0.5; increase BM25 weight for code identifiers, numbers, or domain‑specific terms; increase dense weight for conceptual queries.

Embedding model choice : For Chinese corpora, avoid English‑only models; recommended models include bge-large-zh-v1.5, text-embedding-3-large (supports Chinese), or domestic Acme_Embedding.

Common Pitfalls

Misconception 1 : “Vector retrieval already understands everything, no need for keywords.” In reality, exact identifiers like ERR-4023 are often missed.

Misconception 2 : “Hybrid is just concatenating two lists.” Without RRF or weighted fusion, the merged list is chaotic.

Misconception 3 : “Hybrid is always better.” For purely semantic queries (e.g., “how to improve team efficiency”), dense retrieval may suffice; hybrid shines on diverse query types.

High‑Frequency Follow‑Up Questions

How to choose the RRF smoothing constant k? Default is 60; larger k smooths scores, smaller k emphasizes top ranks.

Does hybrid search add performance overhead? Yes, it runs two searches; mitigate by parallel execution, limiting Top‑K per branch, deduplication, and caching.

Beyond BM25, what sparse methods exist? SPLADE offers lexical expansion with better recall at higher compute cost.

Memory Mnemonic

Hybrid Search Three Steps : Parallel dense + sparse → RRF fusion (rank‑only) → Rerank (Cross‑Encoder). Remember “vectors capture meaning, keywords capture literal, RRF fuses, Rerank refines.”

Summary

Hybrid retrieval’s essence is taking the long of one method and the short of the other : dense vectors excel at semantic understanding but miss exact matches; keyword search nails exact matches but lacks semantic grasp. RRF merges rankings, and an optional cross‑encoder reranks the fused list, forming the production‑grade standard for RAG systems. In interviews, clearly explain *why* hybrid is needed and *how* the fusion works to earn high marks.