Survey of User Representation Learning and Transfer Learning in Recommendation Systems

With the rapid development of large‑scale language models (e.g., BERT, GPT) and vision models (e.g., ResNet, Vision Transformer), the recommendation community has begun to explore self‑supervised pre‑training and transfer learning to obtain universal user representations that can be adapted to downstream tasks such as cross‑domain recommendation and user profiling.

Parameter‑Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation (SIGIR 2020) – Proposed the PeterRec framework, which pre‑trains on user click sequences using self‑supervision and evaluates on five downstream tasks, including cross‑domain transfer and user attribute prediction. Code and dataset: https://github.com/fajieyuan/SIGIR2020_peterrec .

One Person, One Model, One World: Learning Continual User Representation without Forgetting (SIGIR 2021) – Extends PeterRec with a lifelong learning mechanism that prunes redundant parameters to accommodate new tasks while preserving previously learned knowledge. Code and dataset: https://github.com/fajieyuan/SIGIR2021_Conure .

Perceive Your Users in Depth: Learning Universal User Representations from Multiple E‑commerce Tasks (KDD 2018) – Introduces the DUPN model (LSTM backbone) for multi‑task, multi‑objective learning in Alibaba’s e‑commerce scenario, demonstrating that joint modeling of CTR, L2R, PPP, FIFP, and SPP improves user embeddings.

One4all User Representation for Recommender Systems in E‑commerce (arXiv 2021) – Presents ShopperBERT, trained on 8 billion click events, showing state‑of‑the‑art performance on a suite of downstream tasks such as age, profile, and life‑status prediction.

TUM: Pre‑training User Model from Unlabeled User Behaviors via Self‑supervision (EMNLP 2020 Findings) – Proposes mask‑behavior and next‑K‑behavior prediction to learn user embeddings without labels; evaluated on CTR and profiling tasks. Code: https://github.com/wuch15/PTUM .

UPRec: User‑Aware Pre‑training for Recommender Systems (TKDE 2021) – Incorporates heterogeneous user information (e.g., gender) into a pre‑training scheme, addressing data sparsity and demonstrating improvements on user attribute prediction.

User‑specific Adaptive Fine‑tuning for Cross‑domain Recommendations (TKDE 2021) – Introduces a reinforcement‑learning‑based personalized fine‑tuning strategy that decides whether to fine‑tune residual blocks for each user, achieving better performance on cold‑start scenarios.

Scaling Law for Recommendation Models: Towards General‑purpose User Representations (arXiv 2022) – Explores contrastive learning (CLUE) on 500 billion user actions, showing that larger scale yields significant gains across seven downstream tasks.

Learning Transferable User Representations with Sequential Behaviors via Contrastive Pre‑training (ICDM 2021) – Argues that item‑level pre‑training can be sub‑optimal and proposes user‑level contrastive learning to preserve representation quality.

Learning Universal User Representations via Self‑Supervised Lifelong Behaviors Modeling (ICLR 2022 submission) – Introduces LURM, a lifelong learning framework with BoI and SMEN components, aiming to model continuous user behavior streams; code and data are not yet released.

The article also provides additional reading links and encourages readers to follow the WeChat public account for more research updates.