Boosting RAG Retrieval Quality with Cohere Rerank and Cross‑Encoder

1. Why a Rerank Layer Is Needed

Hybrid retrieval (Elasticsearch full‑text + vector search) raises recall, but feeding the top‑20 documents directly to an LLM yields inconsistent answers because only a few of those candidates are truly useful. This causes high token consumption (≈6000 tokens per query) and reduced relevance as the model’s attention diffuses.

The solution is to place a reranker between retrieval and the LLM, reducing the candidate set to the top 3‑5 most relevant documents.

2. Recall vs. Rerank

Recall (Bi‑Encoder) : high coverage, fast ANN search, independent encoding of query and document.

Rerank (Cross‑Encoder) : high precision, joint query‑document encoding, pairwise scoring, slower but more accurate.

These stages trade space for time (recall) and time for accuracy (rerank); they complement each other rather than replace one another.

3. Mainstream Reranker Solutions

Cohere Rerank API

Models: rerank‑v3.5, rerank‑multilingual‑v3.0 Pros: zero‑deployment cost, excellent Chinese support, stable latency (200‑500 ms), one‑line LangChain integration.

Cons: documents are sent to Cohere (no privacy for sensitive data), pay‑per‑search‑unit, requires internet connectivity.

Local Cross‑Encoder (HuggingFace)

Models: BAAI/bge‑reranker‑v2‑m3 (bilingual, strong), cross‑encoder/ms‑marco‑MiniLM‑L‑6‑v2 (English, fast), BAAI/bge‑reranker‑large (stronger but heavy).

Pros: data stays private, no network dependency, can be fine‑tuned on domain data.

Cons: needs GPU for acceptable latency (CPU ≈ 2‑5 s for 20 docs), deployment and service overhead, Chinese performance varies by model choice.

ColBERT v2 (Late Interaction)

Pros: 10‑100× faster than standard Cross‑Encoder, supports end‑to‑end recall‑then‑rerank, good Python integration via RAGatouille.

Cons: large token‑level storage, higher engineering complexity, limited Chinese models.

Jina / Voyage AI Rerankers

API‑based services similar to Cohere; comparable accuracy, slightly different pricing; Voyage excels on English legal/finance corpora, Jina offers solid multilingual support.

Overall comparison (accuracy, latency, integration difficulty) is summarized in the article’s tables.

4. Design Decision: Dual‑Track Strategy

Production environments use the Cohere API because the target C‑end application has no strict privacy constraints, the multilingual model outperforms most open‑source options, and the API eliminates model‑serving overhead (latency 200‑400 ms for 20‑doc rerank).

Development, testing, and offline scenarios use a local Cross‑Encoder to avoid API costs, enable CI/CD in isolated networks, and satisfy data‑privacy requirements.

Both implementations expose the same ContextualCompressionRetriever via a unified factory function, so business code switches between them by changing environment variables only.

5. Core Principle: Why Cross‑Encoder Beats Bi‑Encoder

Query → BERT Encoder → q_vec
Doc   → BERT Encoder → d_vec
score = cosine(q_vec, d_vec)

Bi‑Encoder encodes query and document independently, enabling pre‑computation but lacking token‑level interaction, which hurts precise matching (e.g., exact numbers or dates).

[CLS] + Query + [SEP] + Document + [SEP] → BERT → Linear → score

Cross‑Encoder concatenates query and document, allowing full self‑attention across all tokens; the model can directly relate any query token to any document token, yielding higher relevance scores. The trade‑off is that scoring must be performed for every candidate pair (O(n) complexity), which is why it is applied after a recall step that reduces the candidate set.

6. Implementing Cohere Rerank

Installation:

# TypeScript
npm install @langchain/cohere @langchain/core langchain

# Python
pip install langchain-cohere langchain-openai langchain

Key TypeScript snippet:

import { CohereRerank } from "@langchain/cohere";
import { ContextualCompressionRetriever } from "langchain/retrievers/contextual_compression";

export function createCohereReranker(baseRetriever, topN = 3) {
  const cohereRerank = new CohereRerank({
    apiKey: process.env.COHERE_API_KEY,
    model: "rerank-multilingual-v3.0",
    topN,
  });
  return new ContextualCompressionRetriever({
    baseCompressor: cohereRerank,
    baseRetriever,
  });
}

Python counterpart follows the same logic with CohereRerank from langchain_cohere. Important parameters: topN (usually 3‑5), model (use multilingual version for Chinese), and relevanceScore (0‑1 score returned by Cohere).

7. Implementing a Local Cross‑Encoder

Python class LocalCrossEncoderReranker loads a HuggingFace CrossEncoder, scores (query, doc) pairs, writes scores to relevance_score metadata, sorts, applies an optional threshold, and returns the top‑n documents.

FastAPI service ( rerank/server.py) hosts the model once and exposes a /rerank endpoint that accepts JSON with query, documents, and top_n, returning ranked results.

TypeScript bridge sends a POST request to the service, filters by scoreThreshold, and wraps the selected documents in Document objects.

8. Environment‑Driven Factory

The factory selects the implementation based on COHERE_API_KEY and optional mode flags. Example (TypeScript):

export function createReranker(baseRetriever, config = {}) {
  const { mode = "auto", topN = 3, cohereModel = "rerank-multilingual-v3.0", localServiceUrl = "http://localhost:8765/rerank", scoreThreshold = 0.3 } = config;
  const useCohere = mode === "cohere" || (mode === "auto" && !!process.env.COHERE_API_KEY);
  if (useCohere) {
    console.log(`[Reranker] Using Cohere API (${cohereModel})`);
    return createCohereReranker(baseRetriever, topN);
  }
  console.log(`[Reranker] Using local CrossEncoder (${localServiceUrl})`);
  return createLocalReranker(baseRetriever, { serviceUrl: localServiceUrl, topN, scoreThreshold });
}

Python version mirrors the same logic.

9. Integrating into a LangGraph RAG Pipeline

Both TypeScript and Python pipelines define a state with query, rerankedDocs, and answer. The retrieve node calls reranker.getRelevantDocuments, the generate node builds a prompt with the reranked context and invokes an LLM (e.g., gpt‑4o‑mini). The graph connects START → retrieve → generate → END.

10. Performance Impact

MRR@3 improved from 0.61 to 0.84 (↑38%).

NDCG@3 improved from 0.58 to 0.81.

Average token consumption dropped from ~1800 to ~900.

Query latency increased from ~800 ms to ~1100 ms (≈300 ms extra for reranking).

The modest latency increase is outweighed by the substantial relevance boost and token‑cost reduction, making the rerank layer a cost‑effective upgrade.

11. Conclusions

Hybrid retrieval solves the recall problem; reranking solves the ranking problem—both are essential.

Cross‑Encoder’s joint encoding explains its higher accuracy over Bi‑Encoder.

For Chinese workloads, Cohere’s rerank‑multilingual‑v3.0 offers the best cloud performance with low integration effort.

A dual‑track architecture (Cohere for production, local Cross‑Encoder for development) provides the optimal engineering balance.

Reranking adds 200‑400 ms latency, suitable for scenarios without ultra‑low‑latency requirements; real‑time use cases may need asynchronous pre‑fetching or caching.