How LaSM Pulls GUI Agents’ Attention Back from Deceptive Pop‑up Attacks

GUI agents interpret mobile screens as tasks, but pop‑up dialogs can hijack their attention, causing the agent to click malicious buttons. This “ghost‑hand” phenomenon is the most common and dangerous form of environment‑injection attack because the pop‑up visually resembles the user’s command.

Prior work formalizes these threats as an environment‑interference paradigm and shows that multimodal agents drift toward the pop‑up, producing erroneous action sequences. Existing defenses follow two routes: (1) retraining with negative examples of pop‑up deception, which is costly; (2) prompting the model to ignore pop‑ups, which fails when the pop‑up text is semantically aligned with the task.

The authors propose a third, “engineered‑mechanism” approach that requires no model‑structure changes, no extra inference steps, and no retraining. They introduce LaSM (Layer‑wise Scaling Mechanism), which scales the weights of a few layers once before inference, realigning attention toward task‑relevant regions.

LaSM works by multiplying the attention and MLP projection matrices of selected layers by a coefficient α, modestly amplifying their representations. To identify which layers matter, the paper visualizes layer‑wise attention heatmaps using a training‑independent method. Shallow layers scan layout, middle layers form semantic correspondences, and deep layers focus on candidate buttons that drive the final action.

Two target regions are fixed: the <icon-cross> for closing the pop‑up and the <button-confirm> that the attacker wants the agent to press. For each layer, a local patch around the target pixel is extracted, flattened, and compared across samples using cosine similarity. The similarity trajectories of correct versus incorrect samples diverge in middle layers, revealing a “security‑critical layer” range.

Naïvely scaling all deep layers harms performance, so LaSM performs a layer‑range narrowing search: starting from full‑model scaling, it observes accuracy trends while shrinking the interval, ultimately selecting a middle‑semantic band that improves correct outputs without destabilizing higher‑level aggregation.

The scaling rule is applied as a simple Transformer residual update: for layer l, both the attention sub‑layer output and the MLP sub‑layer output are multiplied by α before proceeding. The scaling is performed on the weight matrices of the four attention projections and the two MLP matrices, not on activations, making LaSM a one‑time weight patch rather than a dynamic inference modification.

Experiments on 12 pop‑up variants (2,400 perturbed screenshots) evaluate defense success rate (DSR). LaSM consistently raises DSR across baselines. For Qwen2‑VL‑7B, baseline DSR under overlay attacks rises from 18.9 % to 66.4 % with LaSM; under inductive attacks, from 14.8 % to 68.3 %. Combining LaSM with chain‑of‑thought prompts approaches 100 % DSR. Similar gains appear for LLaVA‑v1.6‑Vicuna‑13B.

Further analysis defines AttnMean(l) as the average attention intensity on target regions per layer. Scaling the identified critical layer band raises attention on the <icon-cross> and suppresses drift toward the malicious <button-confirm>. Ablation studies show that scaling both attention and MLP weights jointly is essential: scaling only one yields DSR near zero.

α sensitivity experiments (α∈[0.9,1.3]) reveal an optimal range near 1; larger α degrades performance and can produce nonsensical outputs. Real‑world evaluation on an AndroidControl suite inserts synthetic pop‑ups into 224 episodes, yielding 911 images covering normal and attack states. LaSM improves task success rate with minimal impact on normal capability: on OS‑Atlas‑7B‑Pro, Type accuracy drops slightly (97.26 %→94.4 %), Grounding stays stable, while TSR rises from 18.75 % to 30.36 % (≈62 % relative gain).

Failure case analysis shows two edge scenarios: (1) when the screen is otherwise empty, the pop‑up becomes the sole visual anchor and dominates attention; (2) during text input, the model’s TYPE mode focuses on the keyboard layout, ignoring newly appearing pop‑ups. These observations align with recent studies on GUI agent memory shortcuts.

In summary, LaSM does not rely on stronger prompts or additional training data; instead, it exploits the layer‑wise attention drift pattern of GUI agents under pop‑up attacks, identifies a security‑critical layer interval, and applies a lightweight, deployment‑time weight‑scaling patch that pulls attention back to task‑relevant regions, markedly improving robustness in realistic mobile scenarios.