Why the Execution Process Is More Dangerous Than the Final Answer: Evaluating AI Agent Harness Safety with HarnessAudit

AI safety concerns have shifted from model alignment to the execution harness that wraps deployed agents. The harness decides which tools an agent may call, which resources it can access, how information flows between sub‑agents, when execution stops, and how errors are recovered. Consequently, many unsafe failures occur during the execution process rather than in the final answer.

HarnessAudit Framework

Researchers from UCSB introduced HarnessAudit , a security evaluation framework that audits the complete execution trajectory of agents. The framework covers eight real‑world domains (finance, e‑commerce, healthcare, office collaboration, social interaction, daily life, legal compliance, software engineering) and defines 210 tasks for systematic assessment. The paper is titled "Auditing Agent Harness Safety" (arXiv:2605.14271) and the code and datasets are hosted at https://github.com/eric-ai-lab/HarnessAudit.

What HarnessAudit Audits

Boundary Compliance – every tool call, resource access, and inter‑agent communication must follow declared permission and information‑flow policies.

Execution Fidelity – agents must achieve the goal using authorized intermediate steps, without substituting objects, operating on out‑of‑scope resources, or performing actions beyond user authorization.

Stability Under Perturbations – the above properties must hold even when faced with indirect prompt injection, ambiguous goal descriptions, or tool‑call errors.

A trajectory is considered safe only if it passes all three checks; final‑answer correctness is reported separately to measure mismatches between task completion and safe execution.

Key Findings

1. High task‑completion scores do not guarantee safety. In the OpenClaw benchmark, Claude Opus 4.6 achieved a higher task‑completion rate than Gemini 3.1 Pro but received a lower overall safety score because it crossed more safety boundaries during execution.

2. Boundary‑compliance difficulty varies. Tool selection is generally easy – most harnesses choose the correct tool. Failures concentrate after tool selection, especially in resource‑access compliance and information‑flow compliance.

3. Harness design can both improve and amplify risk. For the same Claude model, Claude Code outperformed OpenClaw in both task completion and safety. By contrast, Codex improved completion but reduced safety because GPT‑5.4 in its native environment performed more actions and longer trajectories, accumulating more violations.

Violation Concentration

The most common violation type is resource access : agents call the correct tool but operate on unauthorized objects or files, leading to lower resource‑access compliance than tool‑use compliance.

The second hotspot is inter‑agent information flow . In multi‑agent harnesses, messages are routed correctly but often contain excess context, retain sensitive data after task completion, or leak original data through summaries.

Comparing single‑agent and multi‑agent settings shows a sharp drop in compliance: tool compliance falls from >0.85 to 0.64, resource compliance from >0.85 to 0.63, and information‑flow compliance appears at only 0.58, highlighting the expanded attack surface of collaboration.

Additional Observations

Failures are systemic: over 50% of agents exhibit at least one safety violation on every task, rising to 72% in the OpenClaw configuration.

Longer execution trajectories accumulate more violations and slow down performance.

Risk profiles differ by domain: financial and office tasks mainly suffer resource‑access violations; daily‑life and e‑commerce tasks see information‑flow issues; software‑engineering tasks encounter tool‑use problems.

Stability under perturbations is generally poor; indirect prompt injection reduces stability scores to between 0.15 and 0.22.

Why This Matters Now

Multi‑agent harnesses are becoming the core infrastructure for most serious agent products within the next year, including coding assistants, user‑facing chatbots, and operations agents. Each handoff introduces a risk of leaking information to unintended components. In single‑agent systems the trust boundary is the tool call; in multi‑agent systems the trust boundary shifts to the message bus, which is often treated insufficiently as a security perimeter.

Future Directions

Enforce explicit need‑to‑know policies so agents do not share full context by default.

Base security evaluation on full execution trajectories rather than final answers.

Implement clear need‑to‑know mechanisms for multi‑agent communication, where sub‑agents declare required information and the harness validates permission before transmission.