How Secure Are AI Agents? Risks, Attacks, and Governance Strategies

Overview of AI Agents

AI agents are autonomous systems that perceive an environment, decide on actions, and execute tasks to achieve specified goals. They can understand natural‑language commands, learn user preferences, and decompose objectives into step‑by‑step plans.

Definition

An AI agent perceives its surroundings, decides on actions, and carries out services. It can generate its own prompts to reach a goal (see Figure 1).

LLM‑Powered AI Agents

According to Lilian Weng’s "LLM Powered Autonomous Agents", an LLM‑based agent consists of four modules: the large language model (LLM) as the brain, Memory, Planning Skills, and Tool Use (see Figure 2).

Security and Privacy Risks

Integrating AI agents expands the attack surface, often in ways invisible to users and operators. The principal risks are:

Data leakage : Large volumes of personal and corporate data collected by agents may be exposed through unauthorized or malicious code.

Data sharing and usage : Secure transmission, minimisation, and transparency are required when data is shared with third‑party services.

Model attacks : Adversarial inputs can cause the LLM to produce incorrect or harmful outputs.

Social‑engineering attacks : Crafted language inputs can trick agents into unsafe actions.

Privacy issues : Retrieval‑augmented generation (RAG) and vector databases increase the surface for extracting sensitive information.

Legal and regulatory compliance : Varying data‑protection laws add technical and legal complexity.

User‑Input Risks

Unpredictable or malicious user inputs can lead to unsafe behaviour. Strict input sanitisation and review are essential.

Prompt Injection

Attackers embed malicious prompts that override developer instructions, causing the LLM to follow attacker‑controlled directives. Injection methods include:

Passive injection : Malicious prompts placed in web pages or social posts and retrieved via search.

Active injection : Directly sending crafted prompts via email or API calls.

User‑driven injection : Tricking users into entering harmful prompts.

Hidden injection : Concealing payloads in images, encoded strings, or auxiliary program output.

Jailbreak Attacks

Jailbreaks bypass built‑in safety constraints, allowing the model to generate disallowed content. They are classified as:

White‑box attacks : Exploit internal model information (gradients, logits, fine‑tuning) to craft triggers.

Black‑box attacks : Use only model outputs, employing prompt rewriting, template generation, or fuzzing tools such as EasyJailbreak.

Internal Execution Risks

The core LLM module is a black box, making it vulnerable to backdoor, hallucination, and covert state attacks.

Backdoor Attacks

Attackers inject malicious triggers into training data or model weights, causing targeted misbehaviour when the trigger appears. Four sub‑types are:

Data poisoning : Adding, replacing, or incrementally injecting malicious samples into the training set.

Weight poisoning : Direct modification of model parameters or architecture.

Chain‑of‑thought (CoT) attacks : Inserting hidden malicious reasoning steps into the model’s CoT process.

Hidden‑state attacks : Manipulating intermediate activations to leak information or trigger behaviour.

Hallucination Attacks

Adversaries deliberately provoke the model to generate fabricated information, which can cascade into unsafe decisions. Methods include data injection and model interference.

External‑Entity Interaction Risks

When agents trust external systems, they become vulnerable to memory and context manipulation.

Short‑Term Memory Attacks

Exploiting context‑dependent outputs.

Overloading or exceeding the memory window.

Manipulating hidden‑state vectors.

Embedding triggers or backdoors in training data.

Injecting irrelevant or misleading information.

Long‑Term Memory Attacks

Poisoning training data to embed biased or malicious knowledge.

Planting backdoors that activate on specific triggers.

Data injection that skews model behaviour for extended periods.

Model reverse‑engineering to infer training data.

Misleading the model’s generalisation capabilities.

Governance Recommendations

Clarify agent types : Identify the appropriate agent variant for a task and assess associated risks.

Build AI literacy : Maintain a knowledge base describing agent capabilities, limitations, and safe usage.

Evaluate integration : Verify that an agent’s purpose, scope, and security posture align with organisational requirements before deployment.

Monitor runtime environment : Continuously observe the agent’s operating context to detect anomalies.

Maintain healthy skepticism : Treat agent outputs as untrusted until verified.

Manage dependencies : Keep third‑party libraries and services up‑to‑date and vetted for security.

Secure datasets : Validate data provenance and apply adversarial testing to detect poisoning.

Model security : Use homomorphic encryption, trusted execution environments, and robust access controls.

Deployment safeguards : Harden cloud, edge, and container environments; enforce least‑privilege networking.

Model verification and testing : Conduct thorough adversarial and backdoor testing before release.

Intellectual‑property protection : Guard models and training data against theft.

Auditability : Implement mechanisms to log and review agent decisions and actions.