How Mobile Image Search Powers Real-Time Shopping: Inside Pailitao’s AI Algorithm

Abstract: Mobile visual search has been a generational dream for image‑based search. Since the 1990s, academia and industry have made many efforts. Pailitao’s image search, launched in 2014, has refined algorithms, engineering, and product design through close cross‑department collaboration, resulting in industry‑leading search algorithms.

1. Why do it, why now

1.1 Why

Image‑based search, especially on mobile, is becoming a major traffic entry and user demand. Studies indicate that within five years, over 50% of user intent will be expressed via voice or images. User feedback on Mobile Taobao shows a strong desire for visual search.

1.2 Why now

1. Ubiquitous cameras on mobile devices. 2. Deep learning breakthroughs since 2013 in vision, speech, and NLP. 3. Wide availability of large‑scale computing platforms such as ODPS and Amazon Cloud. 4. Growth of mobile e‑commerce, where users naturally want to “shoot‑and‑buy”, providing abundant data to improve relevance.

2. Algorithm Framework

Pailitao first launched on Mobile Taobao in 2014 with a small entry point. After multiple iterations, a stable five‑module framework emerged:

The modules are Category Prediction, Object Detection, Image Feature Extraction, Retrieval Index, and Ranking. Relevance is mainly handled by the first three modules and ranking, while the index focuses on scalability.

2.1 Category Prediction

Because raw features have limited discriminative power across categories, a category predictor narrows the search space. Pailitao currently covers more than ten top‑level categories and tens of thousands of leaf categories.

2.2 Object Detection

Product images often have complex backgrounds and small objects. Detecting the main object reduces background interference and improves search results, as illustrated by the comparison of queries with and without detection.

2.3 Image Features

Features consist of deep features (CNN) and local features. CNN extracts high‑level semantic information, while local features capture fine‑grained details, complementing each other.

2.4 Retrieval Index

The search process has offline and online stages. Offline, image features are extracted and indexed. Online, a query’s features are extracted and matched against the distributed index for fast retrieval.

2.5 Ranking

Based on multiple image and non‑image features, various optimization functions re‑rank the retrieved results to improve relevance.

3. Pailitao Business

Through continuous iteration, Pailitao now delivers both exact and highly similar product results at massive scale, supporting billions of images across categories such as apparel, bags, shoes, cosmetics, home goods, and accessories. Users can simply take a photo to obtain results without extra steps.

Mobile: Two entry points on Mobile Taobao – a camera button next to the search box and a dedicated Pailitao tab in the Scan page.

PC: Integrated into the PC Taobao search box.

External/web plugins: http://www.pailitao.com

Examples of search results for various categories are shown below.

4. Final Thoughts

Thanks to the entire Pailitao image search algorithm team for their collective achievements. While the system has addressed many user photo‑search needs, there is still room for improvement. The team continues to optimize algorithms, engineering, and product experience to better satisfy users.