Beyond Prompts: Harness-Policy Co-Evolution for Agent Safety by Shanghai AI Lab

Introduction: The Limits of Static Prompt-Based Safety

As LLM agents integrate with browsers, email, databases, and business systems, safety evaluation that only checks final responses for harmful content is insufficient. Agents plan steps, read external information, and invoke tools; risks can arise at any execution stage. Hidden instructions in web pages may alter task goals, email or file content may be misinterpreted as user commands, over-permissive tool policies enable privilege escalation, and erroneous memories or outdated plans can cause multi-turn executions to drift from original authorization. These failures stem from the interplay of the policy model, harness, tools, memory, and environment state.

Fixing safety issues becomes complex: rewriting the entire prompt makes it hard to isolate effective changes; adding refusal rules can impair normal task completion; updating model parameters alone cannot timely address new risks at runtime. The core challenge is converting execution-time failures into reusable, verifiable safety experience.

SHE: Trajectory-Driven Safety Harness Evolution

Shanghai AI Lab, with Fudan University, Shanghai Jiao Tong University, HKUST, and Zhejiang University, introduced SHE (Trajectory-driven Safety Harness Evolution for LLM Agents) and SafeEvolve. Both start from complete execution trajectories.

Modular Harness Decomposition

SHE decomposes the harness into four safety components:

System Prompt : defines overall principles and task boundaries.

Rule Bank : stores structured, reusable rules.

Safety Memory : accumulates recurring failures not yet solidified into stable rules.

Tool Policy : specifies conditions and scope for tool invocations.

This modularity enables localized revision, verification, and rollback.

Evolution Loop from Full Trajectories

Each evolution round begins with a full trajectory: user request, intermediate context, model responses, tool calls, environment feedback, and task outcome. The system diagnoses which step introduced risk, why existing constraints failed, and which component should be updated. For example, if the agent mistakes web-page or email content for user commands, updates target instruction-priority and untrusted-content detection rules; if an unrelated sensitive tool is invoked, the Tool Policy is revised. Cases not yet abstractable as stable rules go into Safety Memory.

Validity and Safety–Utility Verification

Candidate harness versions undergo validity checks and safety–utility verification to prevent superficial gains from over-refusal or capability reduction. An illustrative case: a user only asked for an app introduction, but the agent attempted device-level installation. SHE learned not to block app-related tasks entirely, but to retain introduction, comparison, and official download link capabilities while requiring explicit authorization before installation and forbidding misrepresentation of incomplete actions as completed.

Experimental Results

Experiments on Agent-SafetyBench used 15 tasks for harness evolution and evaluated on the remaining 185 tasks, covering context poisoning, indirect injection, tool tampering, memory injection, and combined attacks. Compared to a static SafeHarness, SHE reduced average attack success rate from 17.1% to 5.5% and raised task usability under attack from 31.6% to 47.6%. On the held-out AgentHarm benchmark, harm score dropped from 19.8% to 9.8%. The evolved safety boundaries transferred to other agent models without re-running the full evolution per model.

SHE establishes a trajectory-to-harness update path, but external constraints do not guarantee the policy model itself understands the same safety boundaries.

SafeEvolve: Distilling Harness Experience into the Policy

When safety knowledge resides only in the harness, the model still relies on outer interception. For risk variants not covered by rules or during longer tool-call chains, early constraints may degrade. SafeEvolve incorporates policy optimization into the evolution loop, converting verified harness experience into the model's own stable decision-making.

Harness-Side: Safety Prompt and Hierarchical SkillBank

SafeEvolve extracts a Safety Prompt and a hierarchical SkillBank from trajectories. The SkillBank holds both general safety principles and task- and tool-specific operational experience, such as detecting untrusted instructions in web pages, verifying attachment permissions before sending email, and requesting user confirmation when authorization is ambiguous. At inference, the system retrieves relevant skills for the current task instead of stuffing all rules into context.

Two-Stage Policy Training

Harness-use SFT : Trajectories collected under the evolved harness are filtered; only samples passing both safety and task-utility checks enter supervised fine-tuning. This teaches the model when to invoke safety skills, how to distinguish user goals from environmental inducements, and how to continue legitimate tasks after blocking dangerous operations.

Harness-augmented RL : With the Safety Prompt and dynamically retrieved skills, the model explores multi-step trajectories. A verifier assesses task completion, hazardous behavior, and tool-call validity. Unlike SFT, RL optimizes the entire trajectory, further constraining "early compliance, late drift" behavior. The desired policy: continue normal tasks; refuse explicitly harmful or unauthorized requests; ignore injected malicious instructions and resume the user's original task; seek confirmation when authorization is unclear.

Results

On Qwen2.5-4B, SafeEvolve reduced AgentDojo attack success rate from 2.37% to 0.79%, increased clean-task utility from 59.79% to 61.86%; on AgentHarm, harm score fell from 56.45 to 12.27, refusal rate rose from 28.98% to 83.83%. On Qwen3-4B, AgentDojo attack success rate dropped from 13.38% to 2.42%, clean-task utility improved from 44.33% to 60.82%, and attack-condition task utility rose from 35.91% to 52.05%.

Three-Step Evolution of Agent Safety

The two works collectively advance agent safety in three steps:

From static safety configuration to trajectory-driven continuous updates. Safety experience moves from post-hoc logs through diagnosis and verification into the next harness evolution round.

From monolithic safety rules to component-level responsibility separation. System Prompt, Rule Bank, Safety Memory, and Tool Policy each own distinct responsibilities, enabling precise localization, verification, and rollback of changes.

From standalone harness updates to harness–policy co-evolution. The harness quickly captures new risks; the policy gradually absorbs them via SFT and RL. The updated policy generates new trajectories that expose gaps for the harness to address.

This path does not imply agent safety is fully automated. Trajectory diagnosis and verifier scoring can still err, and benchmark improvements do not guarantee production robustness. The contribution is shifting safety from a one-time pre-deployment configuration to a system capability that continuously revises itself based on execution experience.

Papers

SHE: Trajectory-driven Safety Harness Evolution for LLM Agents – https://arxiv.org/abs/2608.09885 SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment –

https://arxiv.org/abs/2609.02786