Skip to content
Pandora's Toolbox: Mapping Security Risks Across the AI Agent Lifecycle

Pandora's Toolbox: Mapping Security Risks Across the AI Agent Lifecycle

Peking University and WeChat Security team present a lifecycle-based survey of generalist AI agent security, analyzing how risks propagate across provenance, orchestration, execution, and persistent feedback stages, covering attack surfaces, evaluation benchmarks, and defense layers.

Machine Heart
Machine Heart
Machine Heart

As large language models evolve into generalist agents capable of autonomous skill selection, tool use, file I/O, database access, web browsing, and long-term memory, security concerns shift from output compliance to the entire operational lifecycle. This survey, jointly produced by Peking University's Knowledge Computing Lab (KCL) and Tencent's WeChat Security team, introduces a unified framework that traces risks across four stages: provenance (what capabilities enter the system), orchestration (how latent risks are activated and amplified), execution (how risks become real-world consequences), and persistent feedback (how execution results feed back into future workflows).

Why Component-Level Security Is Insufficient

Traditional research treats model, memory, tools, and environment as isolated components. However, a single attack can span stages: a malicious skill description enters at provenance, influences skill selection during orchestration, triggers dangerous tool calls at execution, and writes polluted content into long-term memory or reusable skills, enabling cross-session persistence and amplification. The paper argues that generalist agent security is fundamentally a lifecycle problem, not a sum of independent component risks.

Lifecycle Stages

Provenance: What Capabilities Are Allowed In?

At this stage the agent reads skill metadata, descriptions, preconditions, and interfaces to build its capability space. Risks include forged provenance, impersonation of trusted components, and hidden threats in natural-language descriptions, scripts, dependencies, configurations, and external resources. Once a dangerous capability passes admission, it can lie dormant until a future task triggers it.

Orchestration: How Latent Risks Are Activated and Amplified

During orchestration the agent interprets user requests, retrieves long-term memory, selects skills, constructs plans, and prepares tool calls. Information from users, memory, and skills converges in a shared context. Benign skills in combination may form privilege-escalation chains; a routine memory lookup can reactivate poisoned content written weeks earlier. Risk begins to alter the agent's decision and control flow.

Execution: How Risks Become Real-World Consequences

The agent interacts with external environments via web browsing, API calls, code execution, file operations, database queries, and computer/mobile control. Environment responses — hidden webpage text, pop-ups, terminal output, search results, MCP tool responses — may carry prompt injections. When the agent treats these observations as high-priority instructions, internal decisions turn into external actions.

Persistent Feedback: Execution Ends, Risk Persists

Execution results can be written into long-term memory, distilled into new skills, or saved as project artifacts. These states re-enter provenance or orchestration in later sessions, creating "reverse contamination." A single security failure may seed the agent's future behavioral evolution.

For example, a malicious skill disguised as a "hotel price comparison assistant" could induce the agent to select a forged booking interface during orchestration, steal itinerary and payment data at execution, and write that interface into long-term memory. Even after the task ends, the risk may re-trigger in subsequent travel tasks.

Four Coupled Attack Surfaces

Skill Supply Chain: Capabilities as Attack Vectors

Skills introduce software supply-chain risks. Threats hide not only in declarative files (e.g., SKILL.md) but also in helper scripts, dependencies, configurations, retrieval databases, and local caches. Benign skills combined may create unauthorized access, privacy leaks, or destructive execution paths. Some attacks rewrite skills within the current session, activating backdoors in future sessions or under specific conditions. SkillHarm [2] studies lifecycle-aware skill attacks that persist across stages.

User-Side Attacks: From Manual Jailbreaks to Automated Red-Teaming

Users can drive agents through harmful requests, role-play, encoding obfuscation, multi-turn manipulation, and optimized jailbreak prompts. Unlike traditional LLM jailbreaks that aim for non-compliant text, agent attacks seek to induce multi-step tool-call attack chains. Methods have evolved from manual templates and heuristic search to agentic red-teamers that plan, reflect, and continuously optimize strategies.

Long-Term Memory: One Poisoning, Many Future Tasks

Long-term memory is automatically retrieved as trusted context. Attackers can directly poison knowledge bases, RAG corpora, and memory indexes, or induce the agent to write false information via conversation, web content, or environment observations. Backdoor memories stay silent until specific topics, queries, or context patterns trigger them.

External Environment: Observations Are Not Just Data

For agents that browse, operate computers, and call tools, environment observations can be misinterpreted as instructions. Malicious webpages, hidden text, GUI pop-ups, file contents, terminal output, API responses, and MCP responses can hijack subsequent reasoning. Attackers need not touch the model directly; controlling what the agent reads suffices to alter its next action.

Security Evaluation: Beyond Final Answers

When risk extends from text to behavior, evaluation must observe tool selection, parameter construction, deviation points, environment injection influence, and actual state changes (file modifications, data leaks, system alterations). The survey organizes benchmarks into three lines:

Domain-specific agent safety benchmarks covering tool use, web browsing, computer and mobile operation.

Generalist agent safety benchmarks focusing on skill-mediated behavior safety and malicious skill detection.

Offline trajectory auditing using full interaction histories or candidate actions to identify risky steps and explain causal chains. AgentAuditor [3] performs trajectory-level safety audits; ToolSafe [4] adds proactive step-level guardrails for tool invocations. Evaluation moves from "did the final task fail?" to "at which step did risk appear, why, and can it be caught before execution?"

Defense Methods: Three Intervention Layers

Based on lifecycle intervention points, defenses are grouped into three layers:

Input and context filtering : skill risk scanning, prompt injection and jailbreak detection, memory write and retrieval guardrails. Goal: identify malicious instructions, anomalous dependencies, forged provenance, and poisoned memory before they enter the decision context.

Decision and control integrity : external context reconstruction, untrusted data isolation, information-flow and tool-dependency constraints; instruction-hierarchy training, preference optimization, and safe tool-use alignment to internalize trust priorities into model policy. Filtering alone is insufficient because benign information combined can still yield dangerous decisions.

Runtime monitoring and execution constraints : pre-execution checks of full plans, single tool calls, or cumulative trajectories, with options to block, correct, request confirmation, or further verify. Safety mechanisms become controllers embedded throughout execution, not just post-hoc judges.

Future: Self-Evolving Agents Require Evolving Security

As agents accumulate experience, update memory, create tools, rewrite skills, and adjust workflows, lifecycle security challenges amplify. Recent work shows self-evolution can permanently encode malicious influence across iterations [5]; capability growth may produce "misevolution" deviating from intended goals [6]; and accumulated experience is not always faithfully understood or used [7]. Static datasets, fixed guardrails, and one-time alignment cannot cover continuously shifting attack surfaces.

The survey proposes a future security loop:

Risk generation : automatically construct executable, verifiable, continuously evolving adversarial environments and safety data.

Risk diagnosis : trace risk sources, critical steps, tool parameters, and causal chains along long-horizon trajectories.

Security evolution : convert new attacks and failure trajectories into training feedback so safety policies update continuously.

In this loop, adversarial generation discovers unknown risks, fine-grained attribution explains how risks form, and continuous security evolution turns diagnoses into new protective capabilities. Future agents must not only self-improve but also continuously discover, diagnose, and repair their own security weaknesses.

Conclusion

Generalist agents connect models, skills, memory, tools, and environments into long-running, continuously accumulating, self-evolving systems. Security risks no longer belong to a single component or interaction. Provenance decides whether malicious capabilities enter; orchestration decides whether polluted information hijacks plans; execution decides whether dangerous actions materialize; persistent state decides whether risks propagate across tasks and sessions. Pandora's Toolbox aims to shift the field from "point vulnerability patching" to "full-lifecycle security governance" — tracking not only where attacks originate but how they propagate, where they manifest, what state they leave, and where security mechanisms should intervene. Agent security should be a control chain spanning capability provenance, decision formation, action execution, and continuous evolution, not just a gate at the output end.

The paper, taxonomy, and continuously updated literature list are publicly available at https://github.com/MurrayTom/Awesome-Agent-Security-Papers and the full text at https://openreview.net/forum?id=A0KU8ja2mo.

Code example

[2] SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction.
[3] AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents.
[4] ToolSafe: Enhancing Tool Invocation Safety of LLM-based Agents via Proactive Step-level Guardrail and Feedback.
[5] Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies.
[6] Your Agent May Misevolve: Emergent Risks in Self-Evolving LLM Agents.
[7] Large Language Model Agents Are Not Always Faithful Self-Evolvers.
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

security evaluationAI agent securityself-evolving agentsgeneralist agentsdefense mechanismsattack surfaceslifecycle securitysecurity survey
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.