May 16, 2025 · Artificial Intelligence

Why Multi‑Turn LLM Evaluation Fails and How a User‑Simulator Can Fix It

The article explains that large language models lose up to 35% performance in multi‑turn conversations, critiques static single‑turn evaluation methods, and proposes a dynamic user‑simulator with loss‑masking techniques to generate realistic test turns and improve assessment reliability.

AI testingLLMRLHF

0 likes · 6 min read

Why Multi‑Turn LLM Evaluation Fails and How a User‑Simulator Can Fix It

DataFunTalk

Sep 19, 2019 · Artificial Intelligence

Alibaba Cloud Xiaomai Dialogue System: Architecture, NLU, Dialogue Management, and User Simulator

This article presents Alibaba's Xiaomai intelligent dialogue platform, detailing its general system architecture, three-tier NLU approaches for zero‑, few‑, and many‑shot scenarios, platform‑centric dialogue management with TaskFlow, robustness and continuous learning mechanisms, and a user simulator for large‑scale data generation and dialogue diagnosis.

dialogue systemmeta-learningnatural language understanding

0 likes · 13 min read

Alibaba Cloud Xiaomai Dialogue System: Architecture, NLU, Dialogue Management, and User Simulator