How to Build a Hybrid Vector‑+‑Text Search with Redis 8 (No GPU Required)

1. Introduction

The author links to three previous posts that already covered data cleaning, vectorization, and index creation. This fourth article records two items: how to run a hybrid retrieval workflow on Redis 8 and the exact steps performed by the hybrid_search function after debug logs are removed.

2. Experimental Environment and Query Input

2.1 Environment

Hardware : two 8‑core, 16 GB Linux servers, no GPU.

Model : Qwen/Qwen3‑Embedding‑0.6B.

Vector dimension : 1024.

Redis version : 8.4.0.

Corpus size : about 43 000 Tang poems (cleaned and converted between traditional and simplified Chinese).

Retrieval strategy : hybrid (vector weight 0.7, text weight 0.3).

2.2 Query Input (Important)

The queries listed are ad‑hoc “smoke tests” used to verify Chinese tokenization, stable Top‑K return with or without keyword hits, and output structure suitability for later reranking.

1. 天涯
2. 故乡
3. 人到中年
4. 在深山中漫步
5. 怀念逝去的故人
6. 壮志未酬
7. 找朋友吃饭
8. 人生艰难
9. 人到中年没有意义
10. 少年要努力
11. 今天天气好啊
12. 多吃点饭
13. 策马奔腾

3. Project Main Flow Overview (Index → Write → Query)

3.1 Index Creation (Key: Chinese language configuration)

The schema defines two TEXT fields for BM25 and a VECTOR field for ANN. Chinese tokenization is enabled by setting index.language = "Chinese".

import redis
from redisvl.index import SearchIndex

def create_index(*, client: redis.Redis, index_name: str, prefix: str, dims: int) -> SearchIndex:
    schema = {
        "index": {
            "name": index_name,
            "prefix": prefix,
            "storage_type": "hash",
            "language": "Chinese",
        },
        "fields": [
            {"name": "author", "type": "text", "attrs": {"weight": 2.0}},
            {"name": "title",  "type": "text", "attrs": {"weight": 1.5}},
            {"name": "text",   "type": "text", "attrs": {"weight": 1.0}},
            {"name": "vector", "type": "vector", "attrs": {
                "dims": dims,
                "algorithm": "hnsw",
                "distance_metric": "cosine",
            }},
        ],
    }
    vl_index = SearchIndex.from_dict(schema, redis_client=client, validate_on_load=True)
    vl_index.create(overwrite=False)
    return vl_index

3.2 Data Ingestion (Hash + Vector bytes)

Each document is stored as a Redis hash containing author, title, text, and the vector serialized to bytes.

import redis, numpy as np
from typing import Dict, List

def add_docs(*, client: redis.Redis, prefix: str, docs: List[Dict]):
    """docs: {id, author, title, text, vector(np.ndarray float32)}"""
    pipe = client.pipeline(transaction=False)
    for d in docs:
        key = f"{prefix}{d['id']}"
        v = np.asarray(d["vector"], dtype=np.float32)
        pipe.hset(key, mapping={
            "author": d.get("author", ""),
            "title":  d.get("title", ""),
            "text":   d.get("text", ""),
            "vector": v.tobytes(),
        })
    pipe.execute()

3.3 Query Main Flow (Embedding → hybrid_search → Top‑K)

The hybrid_search function merges BM25 keyword scores and vector similarity scores into a single hybrid_score. It also performs a second‑stage pure‑vector retrieval when the first round does not produce enough candidates.

def _sanitize_query_text(query_text: str) -> str:
    q = (query_text or "").strip()
    if not q:
        return ""
    # Remove characters that conflict with RediSearch syntax
    q = re.sub(r"[\-\(\)\{\}\[\]\^\"~*?:\\]", "", q).strip()
    # Split on spaces and join with OR (|) if needed
    if " " in q:
        terms = [t.strip() for t in q.split() if t.strip()]
        q = "|".join(terms) if terms else q.replace(" ", "")
    return q

def hybrid_search(*, client, index_name: str, query_vector: np.ndarray, query_text: str,
                  top_k: int = 10, vector_weight: float = 0.7, text_weight: float = 0.3,
                  return_fields: List[str] = None) -> List[Dict[str, Any]]:
    if return_fields is None:
        return_fields = ["text", "author", "title"]
    vec = np.asarray(query_vector, dtype=np.float32)
    sanitized = _sanitize_query_text(query_text)
    # 1) First round: hybrid query (BM25 + KNN)
    query_str = (
        f"@text:{sanitized} =>[KNN {top_k} @vector $vec AS vector_score]" if sanitized
        else f"*=>[KNN {top_k} @vector $vec AS vector_score]"
    )
    q = (
        Query(query_str)
        .return_fields(*return_fields)
        .with_scores()
        .sort_by("vector_score")
        .dialect(2)
        .language("Chinese")
        .paging(0, top_k)
    )
    r = client.ft(index_name).search(q, query_params={"vec": vec.tobytes()})
    results = {}
    for doc in r.docs:
        text_score_raw = float(getattr(doc, "score", 0.0))
        vector_distance = float(getattr(doc, "vector_score", 0.0))
        results[doc.id] = {
            "id": doc.id,
            "text": getattr(doc, "text", ""),
            "metadata": {"author": getattr(doc, "author", ""), "title": getattr(doc, "title", "")},
            "text_score_raw": text_score_raw,
            "vector_distance": vector_distance,
            "has_text_match": bool(sanitized) and text_score_raw > 0,
        }
    # 2) Candidate supplement if needed
    if len(results) < top_k:
        extra_k = top_k + len(results)
        extra_q = (
            Query(f"*=>[KNN {extra_k} @vector $vec AS vector_score]")
            .return_fields(*return_fields)
            .sort_by("vector_score")
            .dialect(2)
            .language("Chinese")
            .paging(0, extra_k)
        )
        extra_r = client.ft(index_name).search(extra_q, query_params={"vec": vec.tobytes()})
        for doc in extra_r.docs:
            if doc.id in results:
                continue
            results[doc.id] = {
                "id": doc.id,
                "text": getattr(doc, "text", ""),
                "metadata": {"author": getattr(doc, "author", ""), "title": getattr(doc, "title", "")},
                "text_score_raw": 0.0,
                "vector_distance": float(getattr(doc, "vector_score", 0.0)),
                "has_text_match": False,
            }
            if len(results) >= top_k:
                break
    # 3) Normalization and hybrid scoring
    text_scores = [x["text_score_raw"] for x in results.values() if x["text_score_raw"] > 0]
    tmin, tmax = (min(text_scores), max(text_scores)) if text_scores else (0.0, 0.0)
    distances = [x["vector_distance"] for x in results.values()]
    dmin, dmax = min(distances), max(distances)
    merged = []
    for x in results.values():
        if x["text_score_raw"] > 0 and tmax > tmin:
            text_norm = (x["text_score_raw"] - tmin) / (tmax - tmin)
        elif x["text_score_raw"] > 0:
            text_norm = 1.0
        else:
            text_norm = 0.0
        if dmax > dmin:
            vector_norm = 1.0 - (x["vector_distance"] - dmin) / (dmax - dmin)
        else:
            vector_norm = 1.0
        hybrid_score = text_weight * text_norm + vector_weight * vector_norm
        merged.append({
            "id": x["id"],
            "text": x["text"],
            "metadata": x["metadata"],
            "source": "both" if x["has_text_match"] else "vector_match",
            "text_score_raw": x["text_score_raw"],
            "vector_distance": x["vector_distance"],
            "hybrid_score": hybrid_score,
        })
    merged.sort(key=lambda item: item["hybrid_score"], reverse=True)
    return merged[:top_k]

4. Result Presentation (Public‑Facing Summary)

The article shows a concise result table that lists, for each query term, the number of returned items, the split between "keyword + vector" and "vector‑only" sources, and the top‑1 author‑title pair. Detailed Top‑5 excerpts for each query are also displayed to illustrate how the hybrid approach surfaces relevant poetry verses.

5. What the hybrid_search Function Actually Does (Engine‑Side Breakdown)

1 Clean the query text : remove characters that could break RediSearch syntax; split on spaces and join terms with | to build an OR expression.

2 First round – hybrid query : combine BM25 keyword filtering with KNN vector similarity in a single RediSearch query.

3 Candidate supplement : if the first round returns fewer than top_k documents, run a pure‑vector KNN query to fill the gap, de‑duplicating results.

4 Normalization + hybrid scoring : min‑max normalize BM25 scores (if any) and convert cosine distances to similarity scores; compute

hybrid_score = text_weight * text_norm + vector_weight * vector_norm

.

5 Sort and return : order by hybrid_score descending, keep only the top‑K, and annotate each hit with a source field indicating whether it came from the first‑round hybrid query ("both") or the supplement stage ("vector_match").

6. Common Pitfall: Query Language Must Be Specified

Setting language = "Chinese" when building the index only configures the index side. The query side also needs an explicit language declaration; otherwise Chinese tokenization fails and the query behaves as a raw term match (e.g., @text:xxx returns an abnormal hit count). The fix is to add .language("Chinese") to both the first‑round and the supplement Query objects.