From BERT to LLM: Language Model Applications in 360 Advertising Recommendation

The presentation introduces the background of 360's advertising recommendation business and explains why language models are needed to address the demand for precise, interest‑driven ad placement across multiple products such as search, lock screens, and information feeds.

It then analyzes the suitability of language models for recommendation systems, outlining three application directions: feature‑extraction encoding, recommendation prediction, and behavior‑sequence modeling, and discusses the evolution from pre‑trained language models (PLM) to large language models (LLM).

In the PLM stage, methods like BERT4Rec and S³‑Rec are described, highlighting their strengths and limitations for ad scenarios, especially regarding item dictionary dynamics and data sparsity.

The LLM stage covers approaches such as KAR, NoteLLM, and generative recommendation frameworks, showing how prompts, multimodal cues, and knowledge injection can enhance feature encoding and prediction, while also noting latency and hallucination challenges.

Practical industrial experiments are detailed, including the construction of a global user representation model using transformer‑based bidirectional attention, incorporation of temporal embeddings, joint training with interest‑label classification, and extensive data cleaning to build a high‑quality behavior dictionary.

Further improvements involve integrating side‑information (search queries, article titles) via BERT embeddings, redesigning sampled softmax computation, and aligning user embeddings with LLM‑generated interest tags through contrastive learning (InfoNCE) and MLP projection.

Finally, the talk summarizes key takeaways—language‑model‑driven training paradigms improve recommendation performance even with modest model sizes, fine‑tuned LLMs provide effective content features, and aligning LLMs with recommendation models opens new avenues for online systems—while outlining future directions such as generative recommendation, multimodal fusion, and interactive user interfaces.