ClawMark: A Living‑World Benchmark for Multi‑Turn, Multi‑Day, Multimodal Coworker Agents

Language‑model agents are increasingly used as co‑workers that assist users across multi‑day workflows, where the environment changes independently (new emails, schedule updates, knowledge‑base edits, multimedia evidence). Existing benchmarks evaluate only single static episodes and text‑only inputs, which is insufficient.

To address this gap, the authors introduce ClawMark, a benchmark for collaborative agents. It features multi‑turn, multi‑day tasks executed in sandbox services whose state evolves between turns. The benchmark includes 100 tasks covering 13 professional scenarios, runs on five stateful services (file system, email, calendar, knowledge base, spreadsheet), and provides 1 537 deterministic Python checkers that score service states without using an LLM‑as‑judge.

The current release evaluates seven state‑of‑the‑art agent systems. The strongest model attains a weighted score of 75.8, yet its strict task‑success rate is only 20.0%, indicating that completing entire end‑to‑end workflows remains rare. Turn‑level analysis shows performance degradation after the first external environment update, highlighting adaptation to changing environments as a key open challenge.

All benchmark data, evaluation framework, and construction pipelines are open‑sourced to enable reproducible research on collaborative agents.