Mastering Redis 8 Vector Search: Indexing, Hybrid Retrieval, and Re‑ranking Techniques

1. Index and Data Structures

The article shows how to leverage Redis 8.4.0 to implement vector recall, keyword filtering/scoring, and candidate re‑ranking on the application side.

1.1 Vector Index and Choice (HNSW / FLAT)

# HNSW (Hierarchical Navigable Small World) index configuration
vector_config = {
    "TYPE": "FLOAT32",
    "DIM": 1024,
    "DISTANCE_METRIC": "COSINE",
    "M": 16,
    "EF_CONSTRUCTION": 200,
}

Redis offers two vector index types: FLAT (exact scan) and HNSW (approximate nearest‑neighbor). The project chooses HNSW for better recall‑speed trade‑off.

Use FLAT when absolute accuracy is required.

Use HNSW for lower latency and adjustable recall, tuning M, EF_CONSTRUCTION and (query side) EF_RUNTIME.

1.2 Index Creation: Schema and Code

The project uses redis-py to create a connection pool and the redisvl (Redis Vector Library) to add vector indexes and perform KNN queries.

schema = {
    "index": {
        "name": "text_vectors_idx",
        "prefix": "doc:",
        "storage_type": "hash",
        "language": "chinese",
        "language_field": "language",
    },
    "fields": [
        {"name": "text", "type": "text"},
        {"name": "metadata", "type": "text"},
        {"name": "vector", "type": "vector", "attrs": {
            "dims": 1024,
            "algorithm": "hnsw",
            "distance_metric": "cosine",
        }},
    ],
}

Key configuration note: language: "chinese" enables Chinese tokenisation; without it Redis would fall back to English tokenisation, breaking Chinese full‑text search.

from redisvl.index import SearchIndex
vl_index = SearchIndex.from_dict(schema, redis_client=client, validate_on_load=True)
vl_index.create(overwrite=recreate)

1.3 Storage Structure: Why Hash

Each document is stored as a Redis Hash with the key prefix doc:. Hashes are simple to write, fields are easily extensible, and they avoid the extra JSONPath handling required by RedisJSON. text: TEXT field for keyword filtering and BM25 scoring. vector: VECTOR field (FLOAT32 bytes) for KNN. metadata: TEXT field storing author, title, dynasty, etc.

2. Redis Retrieval Methods

2.1 Full‑text Search (BM25)

# Full‑text query
query_str = f"(@text:{keyword})"
query = (
    Query(query_str)
    .return_fields("text", "metadata")
    .with_scores()
    .dialect(2)
    .paging(0, top_k)
)
results = client.ft(index_name).search(query)

2.2 Pure Vector Search (KNN)

# Pure vector KNN query
query = (
    Query(f"*=>[KNN {top_k} @vector $vec AS vector_score]")
    .return_fields("text", "metadata", "vector_score")
    .sort_by("vector_score")  # ascending cosine distance
    .dialect(2)
)
results = client.ft(index_name).search(query, query_params={"vec": query_vector.tobytes()})

vector_score

stores cosine distance (smaller = more similar).

Convert distance to similarity with similarity = max(0, 1 - distance) for easier fusion.

2.3 Hybrid Search (Text + Vector)

Hybrid retrieval first filters candidates with BM25, then re‑ranks them using vector similarity.

# Hybrid query example
query_str = (
    f"(@text:{sanitized_query})=>[KNN {top_k * 3} @vector $vec AS vector_score]"
)
query = (
    Query(query_str)
    .return_fields("text", "metadata", "vector_score")
    .with_scores()
    .sort_by("vector_score")
    .dialect(2)
    .paging(0, top_k * 3)
)
results = client.ft(index_name).search(query, query_params={"vec": vec_bytes})

If BM25 returns no hits, the logic falls back to pure vector search to guarantee semantically relevant results.

2.4 Query Cleaning

Remove special characters that have no meaning in poetry search.

For multi‑word queries, split on spaces and join with | (OR) to increase recall in Chinese scenarios.

3. Re‑ranking: Fuse BM25 and Vector Scores

Redis returns two scores per document: text_score from with_scores() (BM25, larger = more relevant) and vector_score (cosine distance, smaller = more similar). The re‑ranking pipeline converts distance to similarity, normalises both scores, applies weights (e.g., 0.3 text, 0.7 vector), and sorts by the fused score.

# Example fusion code
candidates = []
for doc in res.docs:
    text_score = float(doc.score)                # BM25
    vec_dist = float(doc.vector_score)          # cosine distance
    vec_sim = max(0.0, 1.0 - vec_dist)          # similarity
    candidates.append({
        "id": doc.id,
        "text": doc.text,
        "text_score": text_score,
        "vec_dist": vec_dist,
        "vec_sim": vec_sim,
    })
# Normalise and compute fused_score
t_min, t_max = min(x["text_score"] for x in candidates), max(x["text_score"] for x in candidates)
v_min, v_max = min(x["vec_sim"] for x in candidates), max(x["vec_sim"] for x in candidates)

def minmax(x, lo, hi):
    return (x - lo) / (hi - lo) if hi > lo else 0.5

w_t, w_v = 0.3, 0.7
for x in candidates:
    t_norm = minmax(x["text_score"], t_min, t_max)
    v_norm = minmax(x["vec_sim"], v_min, v_max)
    x["fused_score"] = w_t * t_norm + w_v * v_norm
candidates.sort(key=lambda x: x["fused_score"], reverse=True)

4. Optional Engineering Enhancements

4.1 Lua‑based In‑Redis Re‑ranking

For large candidate sets or latency‑sensitive scenarios, move the fusion step into Redis using a Lua script that runs FT.SEARCH, fetches with_scores and vector_score, computes a lightweight hybrid_score, and returns the top‑K IDs (or IDs plus required fields), reducing network round‑trips.

4.2 Extend Schema with TAG Fields for Author/Dynasty Filtering

Introduce author and dynasty as TAG fields for exact filtering. Update the schema, write these fields with each document, and embed filter clauses like @author:{...} @dynasty:{...} in the query string.

def rebuild_index_with_tag_filters(client, index_name: str, doc_prefix: str, dim: int = 1024):
    """Rebuild index adding author and dynasty as TAG fields."""
    schema = {
        "index": {"name": index_name, "prefix": doc_prefix, "storage_type": "hash"},
        "fields": [
            {"name": "text", "type": "text"},
            {"name": "metadata", "type": "text"},
            {"name": "author", "type": "tag"},
            {"name": "dynasty", "type": "tag"},
            {"name": "vector", "type": "vector", "attrs": {"dims": dim, "algorithm": "hnsw", "distance_metric": "cosine"}},
        ],
    }
    index = SearchIndex.from_dict(schema, redis_client=client, validate_on_load=True)
    index.create(overwrite=True)
    return index

def hybrid_search_with_filters(client, index_name: str, author: str, dynasty: str, keyword: str, top_k: int = 10):
    """Hybrid search with exact author/dynasty filtering, keyword BM25, and vector KNN."""
    query_str = (
        f"(@author:{{{author}}} @dynasty:{{{dynasty}}} @text:{keyword})=>[KNN {top_k} @vector $vec AS vector_score]"
    )
    query = (
        Query(query_str)
        .return_fields("text", "author", "dynasty", "vector_score")
        .with_scores()
        .sort_by("vector_score")
        .dialect(2)
        .paging(0, top_k)
    )
    return client.ft(index_name).search(query, query_params={"vec": b"..."})