This article breaks down RAG evaluation into a two‑layer framework, explains the four core metrics—Recall@K, MRR, NDCG, and the four RAGAS scores—shows how to implement them with LangChain.js, highlights common pitfalls, and offers scenario‑specific metric combinations for reliable performance monitoring.
The article explains why retrieval quality dominates RAG performance and outlines a rigorous evaluation framework—including prompt, ranked results, and ground‑truth annotations—and detailed metrics such as Precision, Recall, MAP@K, NDCG@K, MRR, and F‑scores, while discussing chunking strategies, embedding choices, hybrid retrieval, and CI/CD‑driven monitoring to ensure production reliability.
The second‑place team tackled KDD Cup 2020’s Multimodal Recall challenge by fine‑tuning ImageBERT and LXMERT on query‑image pairs, generating negatives, applying AMSoftmax and multi‑similarity losses, ensembling weighted predictions, and using score‑based post‑processing, boosting NDCG@5 to 0.8352 and powering Meituan’s multimodal search pipeline.
This article reviews the Airbnb search ranking paper, detailing offline and online performance gains, the progression from SimpleNN to LambdaRankNN, GBDT/FM NN, and Deep NN models, failed embedding attempts, extensive feature engineering practices, and the production system architecture that enabled large‑scale deep learning deployment.
Suning’s real‑time log analysis system integrates Flume, Kafka, Storm and Spark Streaming to collect, cleanse, and compute metrics like NDCG, ensuring low latency, high throughput, exact‑once processing, and robust data safety while supporting multi‑dimensional analytics on massive online‑offline traffic.
Learning to rank reframes search as a machine‑learning problem that optimizes document ordering rather than numeric prediction, using relevance metrics such as NDCG and feature‑based scoring functions, and comparing point‑wise, pair‑wise (RankSVM) and list‑wise (ListNet) approaches while stressing that proper error definition and feature selection matter more than the specific algorithm.
