Challenges and Considerations of Recommendation Systems: Evaluation, Data Leakage, and the Role of Large Models

The article begins by outlining four main discussion points: the definition of recommendation system problems and the gap between academic and industrial perspectives; offline evaluation methods and typical data leakage problems; data construction issues for recommendation systems; and the positioning of large models within the recommendation model layer.

1. Problem Definition and Academic‑Industry Gap Academic research often relies on static offline datasets (e.g., MovieLens, Amazon, Yelp) that lack real‑time user interaction data, whereas industry systems operate online, collecting continuous interaction logs and optimizing for conversion or revenue rather than pure accuracy metrics such as HitRate or NDCG.

2. Offline Evaluation and Data Leakage Offline experiments aim to predict online performance, but improper data splits can introduce leakage. Five common split strategies are described, ranging from time‑ordered sliding windows (most realistic) to random splits (least realistic). Studies show that many RecSys papers use random or leave‑one‑out splits, which ignore global timelines and can cause future items to appear in the training set, leading to unrealistic recommendations.

Empirical analysis on four datasets (MovieLens‑25M, Yelp, Amazon‑music, Amazon‑electronics) demonstrates that users often leave the platform early while new items continue to be added, confirming the presence of leakage in real data. Experiments with models BPR, NeMF, LightGCN, and SASRec reveal that all models tend to recommend future items that would not exist at test time.

3. Data Construction Issues Using MovieLens as a case study, the article explains that the dataset captures only rating interactions, not the actual time users watch movies. Consequently, MovieLens behaves like a cold‑start dataset and may not reflect online recommendation dynamics.

4. Large Models in Recommendation The discussion shifts to whether large language models (LLMs) can replace traditional recommendation pipelines. While LLMs simplify model design by turning the problem into a prompting task, current offline evaluation metrics may not fully capture their real‑world effectiveness. LLMs have the advantage of fewer engineering constraints but lack access to real user/item attributes and cannot be directly evaluated for online revenue impact.

The article concludes that academic research on LLMs for recommendation provides valuable insights but is limited by offline evaluation shortcomings and the inability to model true online scenarios.