From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless

Multimodal Product Search Solution

Evolution of Search Scenarios

Traditional keyword search cannot satisfy queries that rely on visual appearance or complex natural‑language descriptions. Example scenarios include searching for a uniquely designed hair dryer without knowing its brand, or finding green children’s shorts with a cartoon dinosaur pattern that is not captured by the product title.

General Architecture

The solution is organized into three layers:

Data Processing Layer

Text metadata processing : tokenise title, description, category, tags and index them in a traditional text engine.

Image data processing : a multimodal large model generates descriptive text from the image, which is then tokenised and stored alongside the original text.

Embedding (vectorisation) : both text and image representations are converted into high‑dimensional vectors and stored in a vector engine.

Query & Recall Layer

Text query : keyword matching in the text engine plus semantic matching by converting the query into a vector with the same embedding model.

Image query : the uploaded image is encoded into a query vector and used for nearest‑neighbour search in the vector engine.

Fusion & Ranking Layer

Rerank & Score : results from the text and vector engines are merged and reranked using a combined score that considers both textual relevance and vector similarity.

Result Return : the final Top‑N list is presented to the user.

Key Technologies

Embedding (Vectorisation)

Embedding maps unstructured data (text or images) into structured high‑dimensional vectors.

Dense Model : produces dense vectors where most dimensions are non‑zero (e.g., Word2vec, S‑BERT, LLM‑based models). Dense vectors capture deep semantic similarity.

Sparse Model : generates sparse vectors with only a few non‑zero dimensions (e.g., BM25, SPLADE). Sparse vectors excel at exact term matching.

Hybrid Model : simultaneously outputs a dense and a sparse vector, combining semantic generalisation with precise keyword matching. Hybrid representations consistently outperform single‑type models in benchmarks.

Vector Retrieval

Given a query vector, the system finds the K nearest neighbours (K‑NN) in the vector space.

Euclidean Distance (L2) : straight‑line distance between two points; smaller distance means higher similarity. Normalised to a score via 1 / (1 + L2_norm^2).

Dot Product : sum of element‑wise products; when vectors are L2‑normalised, dot product equals cosine similarity.

Cosine Similarity : measures the angle between vectors; range [-1, 1] with 1 indicating identical direction.

Elasticsearch Vector Support

Elasticsearch provides native vector field types and APIs: dense_vector: stores dense vectors. sparse_vector: stores high‑dimensional sparse vectors efficiently. semantic_text: an abstract type that automatically maps text to the appropriate vector representation via a configured inference model.

Inference API : allows Elasticsearch to call external AI models (e.g., embedding models) in real time during indexing or querying.

Ingest Pipeline : text_embedding or inference processors automatically convert text fields to vectors during data ingestion.

KNN Search : native approximate nearest‑neighbour API on dense_vector fields.

Hybrid Search : combines traditional match queries with KNN vector search in a single request. Score fusion is handled by Reciprocal Rank Fusion (RRF), which ranks documents based on their positions in each result set rather than raw scores.

Performance Optimisation – Quantisation

Vector search on massive datasets is memory‑intensive. Quantisation reduces memory footprint:

Scalar Quantisation (SQ) : maps 32‑bit float32 values to 8‑bit int8 (or 4‑bit) per dimension, achieving up to 4× memory reduction.

Better Binary Quantisation (BBQ) : builds on SQ and further compresses vectors, cutting memory usage by up to 95% while preserving recall with modest degradation.

Example: a dataset of 100 billion 1024‑dimensional float32 vectors (~37 TB) can be reduced to ~1.8 TB using BBQ + HNSW indexing, shrinking required compute nodes from 170 to 9. The trade‑off is a potential recall loss, which can be mitigated by increasing num_candidates in KNN queries.

Best Practices with Elasticsearch Serverless

Overall Practice Architecture

Data resides in an RDS instance (product ID, textual description, image URL). The workflow:

Offline data service extracts records from RDS.

Multimodal vector service calls AI Search Open Platform models (e.g., M2‑Encoder , Qwen2‑VL ) to generate unified text‑image embeddings.

Processed text and vectors are written into Elasticsearch Serverless.

At query time, the front‑end sends a text or image request; the request is vectorised via the AI Search Open Platform and the vector query is sent to Elasticsearch Serverless for multi‑path recall (text + vector). Top‑N results are returned to the user.

Alibaba Cloud AI Search Open Platform

The platform offers a one‑stop enterprise AI search solution:

Support for various data sources (OSS, MySQL, Hudi, Iceberg, MaxCompute).

Search micro‑services for document parsing, multimodal parsing, vectorisation, rerank, LLM inference, and agent orchestration.

Integration with open frameworks (LangChain, LlamaIndex) and compatibility with external vector databases (Milvus, Havenask, Elasticsearch).

Pre‑installed AI models for document/image parsing, embedding, reranking, and generation, with an interactive experience centre for rapid model validation.

Elasticsearch Serverless Features

Serverless Proxy abstracts cluster management, handling authentication, routing, and request rewriting.

Zero‑Ops : no manual capacity planning; users interact with a logical application endpoint.

Pay‑per‑use CU billing with second‑level granularity.

Elastic scaling : automatic resource expansion/reduction based on real‑time load, with adaptive replica and throttling adjustments.

Seamless AI model integration : built‑in Inference API can invoke any AI Search Open Platform model or custom external model via simple API configuration.

Vector optimisations :

Vector fields are excluded from _source by default to save storage and bandwidth.

One‑click enablement of default quantisation strategies ( int8 or BBQ) without manual tuning.

Automatic warm‑up of HNSW and quantised index files to reduce cold‑start latency.

Demo Overview

The end‑to‑end demo walks through extracting product data from RDS, generating multimodal embeddings via the AI Search Open Platform, indexing into Elasticsearch Serverless, and performing real‑time text‑or‑image queries that return accurate product results.

Conclusion

By combining dense, sparse, and hybrid embeddings, native Elasticsearch vector capabilities, advanced quantisation (SQ & BBQ), and the fully managed Elasticsearch Serverless environment, developers can rapidly build scalable, cost‑effective multimodal product search systems that meet modern e‑commerce user expectations.