How Facebook Evaluates Its Newsfeed Recommendations: Metrics, Models, and User Surveys

This article compiles excellent answers from Zhihu contributors Song Yisong and Liu Tao, discussing how Facebook measures the quality of its Newsfeed recommendation and ranking.

1. Machine Learning Models

The core of the recommendation engine is machine learning (supervised learning). Standard academic practices like AUC, feature importance, and model iteration (e.g., more data, different algorithms) are used to assess model quality.

2. Product Data

Even the best models must be validated against product data. Facebook tracks a range of KPIs rather than a single metric, including DAU/MAU, user interactions (likes, comments, shares), post volume, dwell time, revenue, interaction rates, reports and blocks, and detailed content‑type distributions.

For rapid iteration and A/B testing, finer‑grained data are needed, such as content type distribution changes, impact on public accounts, and effects on third‑party platforms.

Long‑term backtests are maintained for major product decisions, e.g., comparing a holdout group without ads to assess ad impact, or a group with chronological feed ordering to evaluate ranking changes.

3. User Surveys

Product data are explicit and passive; user surveys capture subjective quality. Companies like Google and Facebook incorporate user ratings into KPIs, using large‑scale human judgments to evaluate search and recommendation quality.

Key takeaways: never rely on a single KPI, and quantitative metrics can resolve most disputes when KPI limitations are understood.

When using relevance as a metric, models may over‑converge, leading to homogeneous recommendations (e.g., Douban FM example). Balanced metrics that consider both convergence and diversity are essential.

Practical steps: define core metrics (e.g., reading time), decompose into sub‑metrics (article count, average reading time, interaction counts), and run controlled experiments to validate changes against these metrics.

Beware of the difficulty in designing metrics; over‑optimizing a single metric like CTR can incentivize low‑quality content.

In summary, result‑based metrics are preferable to relevance, and careful metric design is crucial.