Google's CLASHarness Benchmarks 15 LLMs on Multi-Agent Conflict in Computer Use

Background: The Need for Multi-Agent Conflict Benchmarks

As AI agents move from solo operation to collaborative computer-use workflows (coding, cloud ops, finance, approvals), existing benchmarks like OSWorld and WebArena assume an ideal premise: environment information is always consistent and all teammate agents are 100% reliable. In reality, sub-agents often report stale cache data, issue contradictory commands, or deadlock shared resources. For example, a sub-agent may claim "production service is healthy, cut traffic now" while a monitoring screenshot shows cascading errors. A principal agent that blindly trusts the text report causes an outage; it must actively open the screenshot to verify ground truth, then roll back config and release shared locks.

CLASHarness: A Controllable, Verifiable Generation Pipeline

To build a scalable conflict testbed containing codebases, terminal logs, multi-agent chat histories, and high-resolution UI screenshots, the team designed a three-layer, eight-stage automated generation and verification pipeline that guarantees 100% task solvability .

1. Stepwise Planning with Local Auto-Correction

The pipeline first plans a global outline and agent roles/permissions, then generates tasks in rolling windows of 3–4 turns to avoid long-context drift. Each turn is decomposed into 8 micro-steps: path/format errors are fixed by code (zero model calls); logical conflicts trigger a retry of only the failed micro-step. This yields a complete 10-turn task package in an average of 95 API calls .

2. Eliminating Text Shortcuts: Critical Evidence Only in 4K Images

To prevent agents from passing by text search alone, CLASHarness defines the ground-truth values needed to resolve each conflict (e.g., actual bill amount, permission matrix state), renders them into 3840×2160 (4K) resolution real UI screenshots or photos, and runs an automatic scanner to purge all answer clues from text files. Agents are forced to inspect images for evidence.

3. Sandbox-Verified Reference Trajectories and Modular Conflict Plugins

All scoring criteria (34.9% exact code checks + 65.1% semantic three-vote majority checks) are extracted directly from reference trajectories that successfully execute in a real sandbox. The framework includes 23 modular conflict plugins so developers can compose new conflict scenarios like building blocks.

The dataset covers 20 real-world domains, 600 multi-turn executable conflict tasks, 7,970 4K visual evidence images, and 3,284 mandatory checkpoints . Paper: https://arxiv.org/abs/2610.19128. Code and data: https://github.com/google-research/CLASHarness.

Evaluation of 15 Frontier Models: Four Counter-Intuitive Findings

The team ran 9,000+ real sandbox trajectories across 600 tasks on 15 models: Claude Opus 5 / 4.8, Sonnet 5, GPT-6 Astra, GPT-5.6 series, Gemini 3.x series, Kimi K3, DeepSeek V4 Pro / Flash, GLM-5.2.

Finding 1: Decent Per-Step Scores (~60%), But No Model Breaks 30% End-to-End Pass Rate

Under a standard 20-step limit, Claude Opus 5 (25.2%) and GPT-5.6 Sol (24.5%) lead; with relaxed steps, GPT-6 Astra reaches 28.7% . Open-source models Kimi K3 (21.8%, 5th overall) and DeepSeek V4 Pro (19.8%) match or exceed several closed-source models. The gap exists because resolving a real conflict requires a complete chain: detect teammate deception → open image to verify ground truth → modify system config → complete handoff/cleanup . Missing any link fails the whole task.

Finding 2: Weak Models Trust Teammate Text; Strong Models Actively Verify via Vision

When a sub-agent's text report conflicts with the environment, weaker models are "easily persuaded" and accept the false text. Stronger models proactively invoke tools to examine monitoring screenshots or bill images, leading weaker models by 31–65 percentage points in visual grounding and evidence citation. Ablation (Figure 4b): disabling image input crashes pass rates by 87–90% across all models — even on tasks without a hard vision requirement, Claude Opus 5 drops from 24.8% to 2.8%.

Finding 3: "Spotting the Conflict" Is Easy; "Fixing and Closing" Is Hard

Comparing two task variants on the same conflict scenes:

Investigate-and-report only: average pass rate 28.1% .

Require hands-on system-state modification: pass rate falls to 11.0% (net drop 18.2 pp).

Full multi-agent collaboration workflow: pass rate plummets to 4.4% .

The closing phase is the biggest failure point: even GPT-6 Astra and Claude Opus 5 fail 58–72% of closing actions . Models often stop after editing the main config file, forgetting to release shared locks, close tickets, or notify downstream sub-agents.

Finding 4: More Thinking Tokens Cause Deadlock Loops; Relaxing Step Limits Unlocks Deep Exploration

In math/coding, more thinking tokens help. In multi-agent interaction, the opposite holds:

Overthinking induces "deadlock loops": Under 20-step cap, higher thinking-token ratio correlates with lower success (ρ = -0.70). Raising Gemini 3.5 Flash from Low to High thinking drops pass rate from 13.5% to 3.0% , cost per solved task soars to $39.54 (55× GPT-5.6 Luna's $0.72). 89.2% of tasks exhaust the 20-step budget without completing the actual work due to internal vacillation and repeated tool calls.

Relaxing step limits enables long-horizon exploration (Figure 4a): For models good at multi-step exploration, the 20-step ceiling is the bottleneck. Expanding to 40 and 60 steps lifts Gemini 3.8 Flash from 12.8% to 29.2% and 30.7% , surpassing all 20-step baselines, while average cost per solved task drops 21% to $3.50 .

Ranking consistency across different generator models (Gemini 3.6 Flash vs. Claude Sonnet 5) is high (ρ = 0.96), confirming benchmark robustness.

Open Source and Outlook: A Data Engine for Multi-Agent Post-Training

Beyond evaluation, CLASHarness is a scalable multi-agent training data synthesis engine . Using 20 domain seeds and 23 conflict plugins, researchers can cheaply batch-synthesize 100%-solvable multimodal conflict trajectories with verifiable reward signals to train next-gen agents that are more reliable, less gullible, and better at collaboration. The generation pipeline, 600 evaluation tasks, and interactive trajectory viewer are now open on GitHub.