How MaFengWo Reduces Position Bias in Its Recommendation Ranking System

MaFengWo's information‑flow recommendation system aims to suggest more enjoyable and useful travel content by processing items through a content pool, recall, ranking, and re‑ranking pipeline.

The ranking stage relies on a CTR‑prediction model trained on exposure‑click data, forming a closed recommendation loop.

User visual attention creates a position bias: items displayed in prominent spots receive higher click rates, which does not necessarily reflect content quality or true user preference, leading to skewed CTR‑based recommendations.

Two main problems arise from position bias: (1) clicks are influenced by display position, giving "good‑position" items higher CTR; (2) high‑CTR items dominate the recommendation loop, crowding out other content.

To mitigate this, MaFengWo incorporates position information as a feature during model training. Since position is known only after ranking, a default value—determined by offline AUC and online A/B testing—is used during prediction. This approach yields modest online changes (CTR −0.3%, per‑user clicks +0.2%, exposure +0.1%, click‑rate −0.1%).

The model jointly trains a ProbSeen component and a pCTR component; only ProbSeen uses position features, so online prediction relies solely on pCTR. This second scheme shows limited gains (CTR +0.9%, per‑user clicks −0.7%, exposure +1.0%, click‑rate −0.3%).

Inverse Propensity Weighting (IPW) is applied by calculating average CTR for each display position, smoothing these rates, and using them as propensity scores to re‑weight training samples. Initial online results show a ~6% rise in content CTR that later fluctuates.

The early‑positive then negative trend of the IPW scheme occurs because weighting ignores that high‑position items are often high‑quality; thus, the method suppresses both position bias and genuine user preference.

MaFengWo defines "position ability" for each slot (see image). This ability is used in two ways: (1) Adjust click features by dividing raw clicks by position ability and recomputing click‑through rates; (2) Use the inverse of position ability as sample weights during training.

Daily updates of position ability, derived from an EM model, lead to noticeable online improvements: CTR +2.1%, per‑user clicks +4.5%, exposure +1.9%, content click‑rate +3.3%.

User behavior data is the lifeblood of recommendation systems, but it contains various biases (exposure, position, popularity, etc.) that can be amplified in the feedback loop. Future work will further investigate these biases to deliver more accurate recommendations.

References

[1] SIGIR 2021 | 广告系统位置偏差的CTR模型优化方案

[2] Guo, Huifeng, et al. “PAL: a position‑bias aware learning framework for CTR prediction in live recommender systems.” Proceedings of the 13th ACM Conference on Recommender Systems. 2019.

[3] Wang, Xuanhui, et al. “Learning to rank with selection bias in personal search.” Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016.

[4] Joachims, Thorsten, Adith Swaminathan, and Tobias Schnabel. “Unbiased learning-to-rank with biased feedback.” Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. 2017.

[5] Ai, Qingyao, et al. “Unbiased learning to rank with unbiased propensity estimation.” The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.

[6] Wang, Xuanhui, et al. “Position bias estimation for unbiased learning to rank in personal search.” Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 2018.

[7] Agarwal, Aman, et al. “Estimating position bias without intrusive interventions.” Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 2019.

[8] Hu, Ziniu, et al. “Unbiased lambdamart: an unbiased pairwise learning-to-rank algorithm.” The World Wide Web Conference. 2019.

[9] Ovaisi, Zohreh, et al. “Correcting for selection bias in learning-to-rank systems.” Proceedings of The Web Conference 2020. 2020.

[10] Yuan, Bowen, et al. “Unbiased Ad click prediction for position-aware advertising systems.” Fourteenth ACM Conference on Recommender Systems. 2020.

[11] Qin, Zhen, et al. “Attribute-based propensity for unbiased learning in recommender systems: Algorithm and case studies.” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.

[12] Chen, Jiawei, et al. “Bias and Debias in Recommender System: A Survey and Future Directions.” arXiv preprint arXiv:2010.03240 (2020).

[13] Cañamares, Rocío, and Pablo Castells. “Should I follow the crowd? A probabilistic analysis of the effectiveness of popularity in recommender systems.” The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018.

[14] Morik, Marco, et al. “Controlling fairness and bias in dynamic learning-to-rank.” Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.