Prompt Compression Creates a New LLM Vulnerability: HKUST’s Black‑Box Attack Framework COMA (ASE 2026)

Large‑language‑model (LLM) agents often need to handle very long contexts, so developers add a Prompt Compression module to condense system prompts, tool descriptions, conversation history, and retrieved documents before feeding them to the model. The authors ask whether this efficiency trick is safe.

Research Background

Traditional attacks on LLM agents (prompt injection, jailbreak, RAG poisoning) assume the malicious payload must appear in the final LLM context. In a prompt‑compressed pipeline the LLM sees only the compressed prompt, meaning the compressor decides which tokens—including safety rules and evidence—are retained.

Adversarial Information Loss (AIL)

The authors define Adversarial Information Loss (AIL) as the ability of an attacker to perturb untrusted inputs (e.g., user requests or external documents) so that the compressor discards critical safety tokens, causing the downstream LLM to violate its original constraints. An illustrative example: a system prompt contains the rule “must never use shell”. An attacker adds a short perturbation to the user request; after compression the negation “never” is dropped, and the LLM executes a shell command it should have refused.

COMA: A Transfer‑Based Black‑Box Attack

Because attackers typically cannot observe the compressor’s parameters or the exact compressed prompt, the paper proposes COMA, a two‑stage optimization framework.

Stage 1: Search the space of possible compressed prompts for a target that would cause the backend LLM to misbehave (e.g., select the wrong tool, answer incorrectly, or ignore a safety rule).

Stage 2: Find a perturbation to the original uncompressed input such that, after passing through a surrogate compressor, the resulting compressed prompt closely matches the Stage 1 target. The perturbation is then tested on the real black‑box pipeline.

Experimental Evaluation

The authors evaluate COMA on three task families—Agent Tool Selection, Question Answering, and System Prompt Corruption—using six widely‑used compressors (both extractive and abstractive). Across 18 settings COMA attains the highest attack success rate (ASR) with an average of 0.71, far surpassing the strongest non‑compression‑aware baseline (ASR 0.21). When the compressor is removed or no attack is applied, ASR drops to ~0.01, confirming that the vulnerability originates from the compression step.

COMA also generalizes: under different compression budgets, model families, and model scales, the average ASR remains around 0.69, showing that swapping the backend LLM does not eliminate the risk because the critical context has already been lost.

Further analysis links the Critical Token Removal Rate directly to ASR, indicating that COMA deliberately removes a small set of behavior‑critical tokens rather than adding random noise.

Real‑World Case Studies

Two concrete pipelines are examined:

In a VSCode Cline scenario, the agent normally refuses to read files outside the workspace. After COMA perturbation, the compressor weakens the “must never read external files” rule, and the LLM proceeds to read a sensitive file.

In a LangChain + Ollama ReAct agent, the correct tool is chosen for code‑feature extraction. COMA’s perturbation shifts the compressed tool description, causing the agent to pick an incorrect tool.

These cases demonstrate that prompt‑compression risks extend beyond offline benchmarks to real software‑engineering agents.

Defensive Strategy: Isolated Compression

Existing defenses such as perplexity‑based detection lose effectiveness against AIL. The authors propose Isolated Compression : separate trusted inputs (system prompts, verified context) from untrusted inputs (user requests, external documents) into distinct compression budgets and add explicit boundary markers when recombining. Experiments show this structural defense restores the safety rule in 96% of cases, because the attacker can no longer crowd out critical tokens.

Implications

The work highlights that efficiency‑oriented components like prompt compressors can silently rewrite the security boundary of LLM agents. Future trustworthy agentic AI systems must analyze not only the backend model but also intermediate stages such as caching, retrieval, compression, and tool orchestration.

Paper: https://arxiv.org/pdf/2510.22963<br/>Code: https://github.com/zsLiu2003/Comattack