Why Most LLM Defense Strategies Fail Against Adaptive Attacks

Background

Three leading AI labs (OpenAI, Anthropic, Google DeepMind) published a paper evaluating the robustness of large‑language‑model (LLM) defenses against jailbreak and prompt‑injection attacks.

Problem

Existing evaluations rely on static attack sets or weak optimizers and do not model an adaptive attacker who can iteratively improve prompts and allocate substantial computational resources.

General Adaptive Attack Framework

The framework treats the attacker as an optimizer and instantiates four typical optimization loops:

Gradient‑based methods that estimate gradients in embedding space and project them back to valid tokens.

Reinforcement‑learning (RL) methods where a policy samples prompts, receives reward from model behavior, and updates via policy‑gradient algorithms (e.g., GRPO).

Search‑based techniques that formulate the problem as combinatorial exploration using heuristics, beam search, genetic operators, or LLM‑guided tree search.

Human red‑team testing that leverages creativity and contextual reasoning, demonstrated through an online competition with >500 participants.

Each loop follows a four‑step cycle: generate candidate prompts → evaluate against the target model (including detector confidence scores) → receive feedback → update the attack strategy.

Experimental Setup

The framework was applied to twelve state‑of‑the‑art LLM defenses covering prompt‑based, adversarial‑training, filtering, and secret‑knowledge categories. Benchmarks such as HarmBench (jailbreak) and AgentDojo (prompt injection) were used alongside the authors’ adaptive tests.

Prompt‑based defenses

Spotlighting

Prompt Sandwiching

Robust Prompt Optimization (RPO)

Static benchmarks report attack success rates (ASR) as low as 1%, but adaptive attacks achieve >95% ASR, and human red‑team tests confirm comparable breach rates.

Adversarial‑training defenses

Circuit Breakers

StruQ

MetaSecAlign

Under adaptive attacks, StruQ and MetaSecAlign suffer 80‑96% ASR, while Circuit Breakers are bypassed with 100% ASR on HarmBench.

Filtering defenses

Protect AI Detector

PromptGuard

PIGuard

Model Armor

Adaptive attacks that exploit confidence feedback achieve >90% ASR against three detectors and 71% ASR against PIGuard.

Secret‑knowledge defenses

Data Sentinel

MELON

RL‑based adaptive attacks achieve >80% ASR; when the attacker models the defense’s internals, ASR rises to 95%.

Conclusions

The study demonstrates that most existing LLM defenses are brittle when faced with strong, adaptive attackers. Robustness claims based on static evaluations are misleading. Future defense research must incorporate adaptive adversaries throughout evaluation and training pipelines to obtain reliable security guarantees.

Paper: The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections (https://arxiv.org/pdf/2510.09023)

Code example

来源：机器之心
本文
约3000字
，建议阅读
5
分钟
本文实测 12 种防御方法，几乎全军覆没。