From Text to Images: Building Multimodal Product Search with Elasticsearch Serverless

With the rapid advancement of AI, users now expect search experiences that go beyond simple keyword matching, allowing queries by image or detailed natural‑language descriptions. Traditional text‑only search struggles to understand visual elements or nuanced product attributes.

The proposed multimodal product search solution consists of three layers: data processing, query & retrieval, and fusion & ranking. Structured product metadata (titles, descriptions, categories) is tokenized and indexed in a traditional text engine, while images are processed by a multimodal AI model to generate descriptive text and high‑dimensional vectors stored in a vector engine.

During a query, textual input is matched against the text engine and simultaneously transformed into a query vector for similarity search in the vector engine. Image queries are directly vectorized and matched against stored image vectors. The results from both engines are merged and re‑ranked using a Rerank module that combines text relevance scores with vector similarity scores.

Key technologies include:

Embedding (vectorization) : dense models (e.g., Word2Vec, S‑BERT, LLM‑based) produce dense vectors that capture semantic similarity; sparse models (e.g., BM25, SPLADE) generate high‑dimensional sparse vectors for exact term matching; hybrid models combine both for optimal performance.

Vector retrieval : similarity is measured using Euclidean distance, dot product, or cosine similarity. Elasticsearch now supports dense_vector and sparse_vector field types, native KNN search, hybrid search, and RRF (Reciprocal Rank Fusion) for robust result fusion.

Quantization : scalar quantization (SQ) reduces 32‑bit float vectors to 8‑bit or 4‑bit integers, cutting memory usage by up to 75 %. BBQ (Better Binary Quantization) further compresses vectors, achieving up to 95 % memory reduction while preserving recall, enabling billion‑scale vector search.

Elasticsearch Serverless provides a fully managed, auto‑scaling backend that abstracts clusters, handles version upgrades, and offers built‑in monitoring. It supports native vector fields, automatic quantization (int8, BBQ), and pre‑warming of HNSW indexes to eliminate cold‑start latency.

The end‑to‑end architecture integrates Alibaba Cloud AI Search Open Platform and Elasticsearch Serverless:

Product data resides in RDS.

An offline data service extracts records and sends them to the AI platform.

The AI platform invokes multimodal models (e.g., M2‑Encoder, Qwen2‑VL) to generate unified vectors.

Processed text and vectors are ingested into Elasticsearch Serverless, stored in separate text and vector indexes.

Online queries from a front‑end are vectorized by the AI platform and routed to Elasticsearch Serverless for combined text and vector recall, followed by Rerank and final Top‑N result delivery.

Elasticsearch Serverless also offers seamless integration with external AI models via the Inference API, automatic resource scaling, pay‑per‑use pricing, and out‑of‑the‑box monitoring dashboards.

Overall, the solution demonstrates how to quickly build a high‑performance, cost‑effective multimodal product search system by leveraging modern embedding techniques, advanced vector quantization, and the managed capabilities of Elasticsearch Serverless.

dense_vector

sparse_vector

semantic_text