How to Evaluate Recommendation Systems: Metrics, Case Study, and Insights

Preface

Recommendation systems are ubiquitous in modern internet products, from e‑commerce suggestions on Taobao to video recommendations on Douyin, shaping user experiences through personalized item lists.

Nature of Recommendation Systems

The concept first appeared in 1990 (Jussi Karlgren) and became a distinct research field by 1994. A widely accepted definition by Resnick and Varian (1997) states that a recommender system provides product information and suggestions to help users decide what to purchase, simulating a sales assistant.

This definition highlights three core questions: how to accurately predict user needs, how to comprehensively describe available information, and how to recommend the most suitable items.

Evaluation Dimensions

Evaluation is typically divided into two major categories: Accuracy (the system’s ability to predict user behavior) and Usefulness , which includes several subjective metrics.

Diversity

Diversity measures the pairwise dissimilarity of recommended items; increasing diversity must not sacrifice relevance to the user’s taste.

Novelty

Novelty reflects how often users encounter items they have not seen before; it is often improved by recommending less popular content while maintaining relevance.

Serendipity

Serendipity captures the system’s ability to surprise users with unexpected yet appealing items, beyond mere novelty.

Trust

Trust indicates the user’s confidence in the system, which can be enhanced by providing explanations or leveraging social connections.

Utility (Real‑time)

Utility assesses whether the recommendation list updates promptly in response to user interactions, which is crucial for time‑sensitive domains such as news.

Evaluation Case Study

The author describes a recent project for 58.com, a platform offering services like recruitment, housing, and used cars. While algorithmic improvements were common, user‑experience evaluation of the recommendation system was lacking.

Instead of the “Case by Case” method (binary Yes/No per item), a quantitative questionnaire was chosen to capture broader dimensions. The rental‑housing business line was selected as the pilot, focusing on the home‑feed scenario.

Results

The evaluation gathered subjective satisfaction scores for accuracy, diversity, novelty, serendipity, trust, and utility across time periods and user segments. These data feed a daily monitoring dashboard, enabling stakeholders to spot weaknesses, investigate low‑scoring users, and provide feedback to the recommendation team.

Reflection

The study identified two main limitations: coarse granularity of evaluation (making it hard to pinpoint problematic items) and high recall burden on users (requiring them to remember past recommendations). Future work suggests real‑time evaluation interfaces that present items instantly for assessment.

References

Wang Guoxia, Liu Heping. “Personalized Recommender Systems Overview.” Computer Engineering & Applications, 2012, 48(7):66‑76.

Paul Resnick, Hal R. Varian. “Recommender Systems.” Communications of the ACM, 1997, 40(3):56‑58.

Xiang Liang. Recommender System Practice . Beijing: People’s Posts and Telecommunications Publishing House, 2012.