Large Models in Recommendation Systems: Evaluation Challenges, Data Leakage, and Practical Considerations

1. Problem definition and industry‑academic gap – Recommendation systems differ between academia and industry in data availability, evaluation metrics, and model usage; academia relies on static offline datasets (e.g., MovieLens, Amazon) while industry works with real‑time user interactions and revenue‑oriented metrics.

2. Offline evaluation and data leakage – Offline tests aim to simulate online performance, but common data splits (time‑based, leave‑one‑out, random) can introduce leakage where future items appear in training, leading to unrealistic performance gains. Empirical analysis on four datasets shows substantial leakage and its impact on ranking results.

3. Data construction issues – Using MovieLens as a case study, the article highlights that the dataset captures only rating interactions, not the actual timing of content consumption, making it a cold‑start proxy rather than a faithful online scenario.

4. Positioning of large models – The discussion shifts to whether large language models can replace traditional recommendation pipelines. While LLMs simplify model design by using prompts, current offline metrics may not reflect real‑world gains, and the lack of user/item attributes limits their applicability.

5. Conclusions – Large models offer flexibility and low deployment cost but lack practical evaluation in live systems; traditional recommendation research remains valuable for understanding data leakage, evaluation protocols, and realistic dataset construction.