Technical Evolution and Architecture of Shenma Search Engine

Shenma Search, launched by Alibaba Group and UC, leverages the massive user base of UC Browser and Alibaba's cloud resources to become a leading mobile search engine in China, achieving a 21.8% market share in the first half of 2018.

The search engine’s relevance technology has progressed from rule‑based methods before 2013, through learning‑to‑rank algorithms after 2013, to deep neural network models in recent years, reflecting a broader industry shift toward machine‑learning‑driven ranking.

Search is presented as a natural AI application because it provides abundant data, real user scenarios, experienced talent, and significant commercial potential, making it an ideal platform for developing and testing AI techniques such as query understanding, semantic matching, and recommendation.

Shenma’s technical stack relies on a distributed framework built on Zookeeper and YARN, ensuring high performance, availability, scalability, and ease of expansion for new features or vertical search domains.

The core modules include query understanding (segmentation, synonym mining, error correction, query expansion), relevance computation (Learning‑to‑Rank, DSSM/CDSSM, BM25, vector recall), and ranking models that combine handcrafted features with machine‑learning predictions (LTR, GBDT, LambdaMart, LightGBM).

Machine‑learning ranking in search differs from traditional ML tasks by focusing on ordering large candidate sets; loss functions such as point‑wise, pair‑wise, and list‑wise are employed, and active learning is used to efficiently label training data.

Deep‑learning solutions transform queries and titles into embeddings via DNN/CNN/RNN/Attention models, compute similarity with cosine or bilinear methods, and train with pairwise or listwise loss functions, mirroring common industry practices.

The article also highlights practical challenges, including handling long‑tail queries with sparse feedback, aligning NLP objectives with ranking goals, and addressing algorithmic, data, and model optimization difficulties in large‑scale production environments.