Build an Image‑Search Engine with Elasticsearch 8.x and CLIP

Reverse image search overview

Reverse image search (also called image‑search) allows a user to upload an image and retrieve visually similar images without typing keywords. Typical use cases include finding duplicate or higher‑resolution versions, locating the original source, and recognizing objects or people in a picture.

Implementation with Elasticsearch 8.x

The solution consists of two core stages: (1) extracting a dense vector representation for each image, and (2) indexing those vectors in Elasticsearch and performing k‑nearest‑neighbors (k‑NN) queries.

Stage 1 – Feature extraction

Use a pre‑trained CLIP model that maps images and multilingual text into a shared embedding space. The recommended model is sentence‑transformers/clip‑ViT‑B‑32‑multilingual‑v1 (OpenAI CLIP‑ViT‑B32 multilingual version). The model can be downloaded from:

https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1

Typical code (Python) to obtain an image embedding:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/clip-ViT-B-32-multilingual-v1')
embedding = model.encode(image)  # <em>image</em> is a loaded PIL or NumPy array

The resulting vector is a 512‑dimensional float array, e.g.:

[-0.72455883, 0.01825839, -0.14531010, -0.08420199, ...]

Stage 2 – Indexing and k‑NN search

Store each embedding in an Elasticsearch index using the dense_vector field type. A minimal index mapping example:

{
  "mappings": {
    "properties": {
      "image_id": {"type": "keyword"},
      "image_name": {"type": "keyword"},
      "relative_path": {"type": "keyword"},
      "image_embedding": {"type": "dense_vector", "dims": 512}
    }
  }
}

After indexing, perform a k‑NN query with the Elasticsearch k‑NN plugin. Example request that returns the top 5 most similar images:

POST my-image-embeddings/_search
{
  "knn": {
    "field": "image_embedding",
    "k": 5,
    "num_candidates": 10,
    "query_vector": [-0.72455883, 0.01825839, -0.14531010, -0.08420199, ...]
  },
  "_source": ["image_id", "image_name", "relative_path"]
}

Key parameters:

field : name of the dense_vector field.

k : number of nearest neighbours to return.

num_candidates : size of the candidate set examined by the engine (larger values improve recall at the cost of latency).

query_vector : the embedding of the query image.

Typical system architecture

Data layer : raw images collected from the web or internal sources.

Collection layer : crawlers or existing tools download images to local storage.

Storage layer : each image is passed through the CLIP model, the resulting vector is indexed in Elasticsearch.

Business layer : a REST endpoint receives a query image, extracts its vector, and issues the k‑NN request to retrieve similar images.

Summary

The pipeline relies on two components: (1) a pre‑trained multilingual CLIP model for high‑quality image embeddings, and (2) Elasticsearch 8.x with the k‑NN plugin for scalable vector storage and fast nearest‑neighbour retrieval. This combination enables a language‑agnostic reverse image search service.

References

https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1

https://github.com/rkouye/es-clip-image-search

https://github.com/radoondas/flask-elastic-image-search

https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html

https://unsplash.com/data