SafeEvolve: Co-Evolving Harness and Policy for Self-Improving Agent Safety

As LLM agents gain autonomous planning, tool use, web access, and multi-step execution capabilities, agent safety faces new challenges. Traditional safety alignment focuses on final output harmfulness, but for agents interacting continuously with environments, risks can arise in intermediate decisions, tool calls, and environment information processing. Malicious instructions in web pages, hidden content in files, and untrusted tool results can hijack agent goals, leading to data leaks or unauthorized actions. Therefore, agent safety depends on both model parameters and the runtime Harness.

Limitations of Existing Approaches

Current safety alignment methods fall into two categories:

Harness-only updates : Safety prompts, rule libraries, skills, and runtime guards. Advantages include fast modification, transparency, and easy rollback. However, as safety rules grow complex, a frozen Policy may fail to understand and execute them accurately.

Policy-only updates : SFT, preference optimization, or RL to bake safety into model weights. But if Policy trains only in a fixed Harness, it cannot timely absorb new deployment risks, and learned safety may not transfer to other agent systems.

SafeEvolve: Harness–Policy Co-Evolution from Execution Experience

SafeEvolve connects both lines, letting Harness and Policy co-evolve from the same batch of agent execution experience.

Trajectory-Driven Harness Evolution

SafeEvolve first collects complete execution trajectories from agent–environment interactions, recording task goals, environment observations, tool calls, risk types, task completion, and safety verification results. When a safety failure is detected for a task class, the system analyzes the failure location and cause, converting the experience into a Safety Prompt or a localized update in a hierarchical SkillBank. These updates are not deployed immediately. Instead, the system modifies only a limited set of safety components and runs paired evaluations: the updated and previous Harness versions are tested on the same tasks, in the same environment, with the same Policy. Only candidates that simultaneously satisfy safety, task utility, and execution quality requirements are accepted and saved as versioned safety assets with evidence logs and rollback conditions.

Teaching Policy to Use Safety Experience

After Harness evolution, SafeEvolve updates Policy via a two-stage SFT–RL pipeline:

Harness-use SFT : The model learns under the evolved Safety Prompt and SkillBank to recognize relevant safety experience, ignore irrelevant information, and correctly execute safety skills during multi-step tool calling.

Harness-augmented RL : The model continues multi-step exploration in the evolved Harness. A Verifier evaluates task completion, dangerous behavior occurrence, and tool-call validity, constructing a safety–utility reward signal.

The optimized Policy again interacts with the environment, generating new execution trajectories that become the safety experience for the next round of Harness updates and Policy optimization, forming a continuous closed loop.

Experimental Results

Experiments on Qwen3.5-4B and Qwen3-4B demonstrate significant improvements:

Qwen3.5-4B on AgentDojo : Attack success rate (ASR) dropped from 2.37% to 0.79% (≈1/3), while clean task utility rose from 59.79% to 61.86%.

Qwen3.5-4B on AgentHarm : Harm score fell from 56.45 to 12.27; safety refusal rate increased from 28.98% to 83.83%.

Qwen3-4B on AgentDojo : ASR decreased from 13.38% to 2.42%; task utility improved from 44.33% to 60.82%; task utility under attack rose from 35.91% to 52.05%.

These results show that letting the external Harness rapidly absorb new risks and then gradually internalizing safety experience into the Policy via training achieves a better balance between safety and task capability.

A New Paradigm for Agent Safety

SafeEvolve explores a new agent safety paradigm: safety mechanisms are no longer static rules configured before deployment, and models are no longer trained once for safety. Instead, Harness and Policy co-learn from real execution experience, continuously improving agent safety behavior in complex environments.

Paper: https://arxiv.org/abs/2609.02786 Code:

https://github.com/MaoPopovich/SafeEvolve