Telepath: A Vision‑Based Recommender Model Inspired by Human Visual Perception

AAAI (Association for the Advancement of Artificial Intelligence) is a top‑tier AI conference recommended by the China Computer Federation (CCF) as an A‑class venue. The 32nd AAAI conference was held in New Orleans, USA, from February 2‑7, 2018.

Jingdong’s strategic "Boundary‑less Retail" initiative calls for intelligent retail infrastructure, and recommendation systems are a core technology. JD’s Intelligent Advertising Lab, led by Vice President Yan Weipeng, focuses on deep learning, reinforcement learning, NLP, and computer vision. Their latest achievement is the Telepath recommendation model, whose paper was accepted at AAAI 2018.

Telepath: Independent Development and Deployment

Telepath is a vision‑based bionic recommender that mimics human brain activity during shopping decisions, aiming to understand users from a visual perspective. The model has been deployed in JD’s recommendation and feed‑advertising scenarios, delivering notable online performance improvements.

Model Overview

The system follows the classic two‑stage recommendation pipeline: retrieval (selecting a few hundred candidates from billions of items) and ranking (scoring candidates to decide final display). Telepath focuses on the ranking stage, but its techniques are applicable to retrieval as well.

Figure 1  Recommendation System Architecture

Telepath Architecture

The model consists of three components inspired by the human visual system:

Vision Extraction: a convolutional network (a lightweight Inception variant) transforms product images from the user’s browsing sequence and candidate set into activation signals.

Interest Understanding: DNN and RNN branches process the activation signals to capture indirect (subconscious) and direct (conscious) user interests, respectively.

Scoring: a standard DNN combines user interest vectors with candidate activations to predict click/purchase likelihood.

An auxiliary Wide & Deep‑style branch further enhances expressive power.

Figure 2  Telepath Model Structure

Visualization

t‑SNE visualizations of the Vision Extraction and Interest Understanding modules show that the network can distinguish different product categories and, to a lesser extent, different user interest groups, confirming the model’s ability to capture visual and interest signals.

Figure 3  Visualization of Vision Extraction Activations

Figure 4  Visualization of Interest Understanding Activations

Experiments

Offline comparisons against a Wide & Deep baseline show lower loss and higher AUC for Telepath. Online A/B tests in a JD app recommendation slot (Table 1) and a JD partner advertising channel (Table 2) report consistent lifts in click‑through rate (CTR), gross merchandise volume (GMV), orders, and return on investment (ROI) across multiple days.

Date Day1 Day2 Day3 Day4 CTR +0.02% +2.37% +1.93% +2.84% GMV +15.04% +7.81% -2.36% +10.05% Orders +6.62% +5.10% +8.54% +13.92% Table 1  Telepath Online Performance in a JD App Recommendation Slot Date Day1 Day2 Day3 Day4 CTR +5.15% +8.07% +10.5% +6.15% GMV +126.48% +9.1% +18.4% -19.24% ROI +129.53% +14.35% +14.2% -17.44% Table 2  Telepath Online Performance in a JD Partner Advertising Channel Telepath has been continuously evolved and applied across JD’s recommendation and advertising pipelines, demonstrating that vision‑driven user modeling can yield substantial business impact. Conclusion The model will keep advancing to serve JD’s billions of users, and the research team plans to release further results in due course. The work exemplifies how foundational AI research can be translated into large‑scale e‑commerce applications. Contact for academic exchange: [email protected]