From Text to Images: Building Multi‑Modal Product Search with Elasticsearch Serverless

Search Scenario Evolution

Traditional keyword search works well for queries like “dress” or “Nike Shibuya limited edition,” but it cannot satisfy visual queries such as searching for a product by a photo of a unique hair‑dryer or a green children’s shorts with a cartoon dinosaur. Users increasingly expect to retrieve items using images or natural‑language descriptions of complex scenes.

General Multi‑Modal Product Search Architecture

The solution consists of three layers: data processing, query & recall, and fusion & ranking.

Data Processing Layer

Text metadata processing : raw product titles, descriptions, categories, and tags are tokenized and indexed in a traditional text engine.

Image processing : a deep CNN (e.g., ResNet) extracts visual features from the image and converts them into a dense vector.

Embedding : both text and image vectors are stored in a vector engine.

Each product therefore has a textual representation in the text engine and a high‑dimensional vector in the vector engine.

Embedding Technologies

Two major families of embedding models are used:

Dense models (e.g., Word2Vec, S‑BERT, large‑language‑model based encoders) produce dense vectors where most dimensions are non‑zero, capturing deep semantic similarity.

Sparse models (e.g., BM25, SPLADE) generate high‑dimensional sparse vectors with only a few non‑zero entries, preserving exact term matching.

Hybrid models combine dense and sparse vectors to obtain both semantic generalization and precise keyword matching, yielding superior performance on benchmark tests.

Vector Retrieval

Similarity is measured using several distance functions:

Euclidean distance (L2) : the straight‑line distance between two vectors; smaller distance means higher similarity.

Dot product : for normalized vectors it is equivalent to cosine similarity.

Cosine similarity : the cosine of the angle between vectors, ranging from –1 to 1.

Scores are often normalized, for example using the formula 1 / (1 + L2_norm^2) so that a smaller distance yields a higher score.

Elasticsearch Vector Support

Elasticsearch provides native field types for vector storage: dense_vector: stores dense float vectors. sparse_vector: stores high‑dimensional sparse vectors efficiently. semantic_text: an abstract type that automatically maps text to the appropriate vector type via a configured inference model.

Ingestion pipelines can invoke the text_embedding or inference processors to convert raw fields into vectors on the fly.

Hybrid search combines traditional match queries with KNN vector search, and result fusion is handled by Reciprocal Rank Fusion (RRF), which ranks documents based on their positions in multiple recall lists rather than raw scores.

Vector Quantization for Performance Optimization

High‑dimensional float32 vectors (e.g., 1024‑dimensional) consume large memory. Quantization reduces memory footprint:

Scalar Quantization (SQ) : maps each float32 value to an int8 (1 byte) or int4 (0.5 byte) representation, achieving a 4×–8× reduction.

Better Binary Quantization (BBQ) : builds on SQ and can cut memory usage by up to 95 %, making billion‑scale vector search feasible. The trade‑off is a modest recall loss, which can be mitigated by increasing the num_candidates parameter.

Example: a dataset of 100 billion 1024‑dimensional float32 vectors would require ~37 TB of RAM; after BBQ quantization plus HNSW indexing, the requirement drops to ~1.8 TB, reducing the required compute nodes from 170 to 9.

End‑to‑End Practice with Alibaba Cloud AI Search Platform & Elasticsearch Serverless

The workflow is:

Product data (ID, textual description, image URL) resides in an RDS instance.

An offline data service extracts the records.

The AI Search Open Platform invokes built‑in models (e.g., M2‑Encoder, Qwen2‑VL) to generate multi‑modal embeddings.

Processed text and vectors are written to Elasticsearch Serverless, which stores both a text index and a vector index.

At query time, a front‑end sends a text or image request; the AI Search platform vectorizes the query, which is then sent to Elasticsearch Serverless for combined text‑plus‑vector recall.

Elasticsearch returns the top‑N results, which are presented to the user.

Elasticsearch Serverless Highlights

Serverless abstracts away clusters, nodes, and shards, offering:

Zero‑ops management : automatic upgrades, security patches, and scaling.

Pay‑as‑you‑go pricing based on Compute Units (CU) with second‑level granularity.

Elastic scaling : resources expand or shrink automatically according to load.

Seamless AI model integration via the Inference API; users can also plug in custom models through simple API configuration.

Vector‑specific optimizations : automatic exclusion of vector fields from _source, one‑click activation of int8 or BBQ quantization, and auto‑pre‑warming of HNSW and quantized indexes to reduce cold‑start latency.

Conclusion

By combining dense, sparse, and hybrid embeddings, advanced vector similarity metrics, and aggressive quantization, developers can build highly accurate multi‑modal product search systems. Leveraging Alibaba Cloud’s AI Search Open Platform together with Elasticsearch Serverless provides a fully managed, cost‑effective, and scalable backbone that eliminates operational overhead while supporting custom or built‑in AI models.