Meituan LongCat’s VitaBench 2.0: A New Benchmark for Long‑Term Dynamic Agents

VitaBench 2.0 is the first benchmark that assesses agents in real‑life, long‑term dynamic user modeling scenarios, extending the earlier VitaBench 1.0 released in October 2022.

The benchmark combines rich user ecosystems with complex tasks: it features 56 simulated users , each with a unique identity derived from real‑world statistics, 819 executable tasks spanning the users’ lifecycles, over 2 000 dynamic preferences , and 66 usable tools . On average each user generates 2 093 interaction events over a time span of 1 580 days (maximum 2 974 days ), faithfully reproducing preference evolution and drift.

VitaBench 2.0 provides a unified evaluation platform for both in‑context learning and memory strategies , allowing a direct comparison of two representative memory mechanisms:

Agentic Memory : the model decides what to retain or forget, maintaining a concise user profile.

RAG Memory : an external retrieval system fetches the most relevant historical fragments for the current task.

Experiments show that adding memory does not automatically improve performance. In the “open‑book” setting where the full history is visible, the strongest model, Claude‑Opus‑4.6 , achieves an average score just above 0.5. Switching to realistic memory modes leads to noticeable score drops, indicating that memory must be correctly updated, retrieved, and utilized.

Performance also degrades over time: as the task sequence index increases, the average scores of all models decline, reflecting limitations in handling ultra‑long contexts and accumulating memory errors.

Enabling the models’ “thinking mode” does not consistently help personalization; many models perform worse when the mode is on. Moreover, models generally lack the ability to ask clarifying questions in proactive tasks—referred to as “eye‑sight”—resulting in sharp performance cliffs (e.g., the Claude family drops from 46.0 to 27.4).

Even when the true user preferences are directly supplied, most models still fail, demonstrating that correctly applying accurate user profiles under high‑pressure, multi‑constraint decisions remains a major challenge.

Failure‑mode analysis reveals two dominant error classes: early models mainly make tool‑usage mistakes (Class A), while stronger models such as DeepSeek‑V4‑Pro primarily stumble on preference understanding and application (Class B). This highlights personalization as the current bottleneck for agents.

Overall, VitaBench 2.0 pushes the evaluation paradigm forward—from single‑shot tasks to long‑term companionship, from passive execution to proactive communication, and from opaque black‑box testing to transparent, data‑driven analysis—offering a concrete bridge between technical research and product development.

VitaBench 2.0 is now open‑source, inviting models to take the “emotional intelligence” challenge.