Search and Recommendation Algorithms: Evolution, Common Pipelines, and Integrated Engine Design

Search and recommendation systems are ubiquitous and aim at personalization. Recommendation establishes a matching relationship between users and items, while search adds a query dimension to the matching.

Evolution Roadmap : The solution evolves from manual operations and hot‑list displays (no user data) to simple models (stage 2) and finally to complex models that fully exploit abundant data (stage 3). The progression follows the same pattern across scenarios.

Typical Pipeline : The overall flow consists of a recall layer, a ranking layer, and a re‑ranking layer. Data (real‑time, near‑real‑time, or offline) is transformed by feature engineering into user, item, and context features, which serve as inputs for the models.

Recall Algorithms : Recall reduces the load on downstream ranking and guarantees relevance. Common personalized recall methods include hot‑list, LBS, u2tag2i, collaborative filtering, and new‑item recall. Modern recall relies on embedding‑based frameworks (e.g., YouTube, Airbnb). Vector similarity is computed using engines such as Milvus. Query parsing (pre‑processing, tokenization, spelling correction, entity recognition) is performed with BERT‑based models to ensure textual relevance.

Fine‑Ranking Model Evolution : Fine‑ranking combines large‑scale discrete and continuous features with sophisticated models. Early stages used small DNNs; later stages adopted large‑scale discrete DNNs such as DeepFM, DCN, DIN. As scenarios grew more complex, multi‑task models like ESMM, MMOE, MBN and meta‑learning approaches were introduced.

Re‑Ranking Algorithms : Re‑ranking improves user experience and content diversity while respecting business rules (e.g., item shuffling, exposure filtering, new‑item boosting). The goal is to adjust model outputs without drastic changes, balancing user satisfaction and operational constraints.

Integrated Search‑Recommendation Engine : To address the high cost of building separate pipelines for each new business scenario, an integrated engine and algorithm component pool were proposed. This design merges common modules (recall, ranking, content understanding) and provides a low‑code integration path.

Data Standardization : Offline data is mapped to standardized fields; online events (exposure, click, order) are normalized via unified logging. Standardized features become the consistent input for all algorithm components.

Configuration Center : Offline SQL mappings generate standard user/item features. The configuration center orchestrates the online pipeline (QP, recall, ranking, re‑ranking) and allows new business systems to plug in with minimal effort.

Intelligent Algorithm Components : The component pool includes intent‑recognition (tokenization, entity extraction), recall (hot, tag, query, item2vec), ranking (DeepFM, XDeepFM, MBN), and re‑ranking (boosting, exposure filtering, shuffling). These components enable rapid deployment across many scenarios.

Applications at 哈啰 : The integrated engine has been applied to over ten scenarios, including product recommendation (finance, e‑commerce, electric‑bike marketplace), feed‑type recommendation (community, user‑activity streams), and search ranking (rental, hotel). Significant improvements in CTR, GMV, and user engagement have been observed.