Hybrid Retrieval in RAG: Combining BM25 Precision with Dense Vector Semantics

BM25

BM25 is essential for scenarios requiring strict lexical matching such as error codes, product models, and domain‑specific terms. It runs on inverted indexes and provides deterministic results, traceable relevance scores, and exact token matching unaffected by semantic blurring.

Determinism : Same query on the same index version yields identical results, independent of model weights or random seeds.

Score traceability : Each match can be broken down into which terms hit which document fields and the contribution of each term’s IDF and term frequency.

Lexical exact matching : Handles abbreviations, product codes, error identifiers, function names accurately.

Retrieval Strategies

Choosing between pure vector retrieval and hybrid retrieval depends on corpus characteristics, query distribution, and system constraints.

Pure Vector Retrieval

Corpus consists mainly of natural‑language narratives with low density of domain terms, abbreviations, or exact identifiers.

User queries are primarily conceptual.

No hard requirement for explainability.

Business cost of wrong answers is low.

In early validation phases, engineering complexity must be limited.

Typical use cases include content recommendation and open‑domain question answering.

Hybrid Retrieval

Corpus contains controlled‑vocabulary‑dense technical documents, legal contracts, medical records, or financial reports where precise terminology matters.

User queries mix semantic questions with exact identifier lookups.

System demands strict accuracy; errors have real business or legal consequences.

Auditability is required; the system must explain the basis of retrieval decisions.

Hybrid Retrieval Architecture

Fusion Strategies

Hybrid retrieval runs BM25 lexical search and dense vector search in parallel, producing two independent ranking lists that are merged by a fusion algorithm. Two main fusion strategies differ in engineering trade‑offs.

Reciprocal Rank Fusion (RRF) converts each candidate’s rank in a list into a score: score = 1 / (k + rank), where k is a constant (commonly 60) and rank is the document’s position. Elasticsearch 8.x, OpenSearch, Weaviate, Qdrant and other major engines support it natively.

RRF is highly robust to score calibration errors because it operates in rank space rather than raw score space, avoiding the difficulty of comparing BM25 scores, cosine similarities, or different embedding similarity distributions. Its limitation is loss of score magnitude information; documents with very different similarity values receive identical contribution if they share the same rank.

Linear (Convex) Combination retains score magnitude by linearly weighting the two scores: final = α·score_BM25 + (1‑α)·score_vector. The parameter α directly controls the contribution of lexical retrieval; larger values give BM25 dominance. After collecting a modest set of high‑quality query‑relevance pairs, tuning this single parameter consistently outperforms RRF on both in‑domain and out‑of‑domain evaluations. Normalization (min‑max or z‑score) is a secondary concern, with little impact between linear methods.

Externalizing the Retrieval Pipeline

OpenSearch and Elasticsearch can natively run BM25 and vector search in a single request and merge results, but the built‑in scorer mixes the two signals into a single score, making each component’s contribution unobservable. Moreover, the normalization, weighting, and re‑ranking logic are locked inside the engine’s DSL, preventing custom business rules in application code.

Externalizing BM25 and vector retrieval as separate steps, then performing fusion in application code, offers several benefits:

Each mode’s scores and hit information can be observed independently.

Fusion logic, weight adjustments, and business rules are managed in regular code, free from engine constraints.

Supports custom RRF weights, intent‑aware routing, and multi‑stage retrieval pipelines.

The trade‑off is increased system complexity. For scenarios without strict audit requirements and in rapid‑validation phases, the all‑in‑one approach may be preferable. For enterprise use cases demanding fine‑grained control and compliance, externalizing the fusion is often necessary.

A Sample Retrieval Pipeline

Conclusion

Hybrid retrieval combines the complementary strengths of lexical BM25 and dense vector search rather than simply stacking them. BM25 supplies lexical precision and traceable scoring; dense vectors provide semantic coverage. Externalizing the fusion grants fine‑grained control over each component’s contribution, addressing the limitations of pure vector retrieval in precision and auditability.