How AdaVD Achieves Precise, Fast, Low-Cost Concept Erasure in Diffusion Models

Background

Rapid advances in text‑to‑image diffusion models enable users to generate high‑quality images from simple prompts, but they also facilitate misuse such as plagiarism of artistic styles, defamation of IP characters, and creation of unsafe content. A practical solution must (1) completely remove the target concept from the generated image and (2) preserve unrelated semantics, while being low‑cost and fast enough for real‑time services.

Limitations of Existing Concept‑Erasure Methods

Existing methods fall into two categories:

Training‑based approaches fine‑tune the diffusion model’s parameters. They achieve strong erasure but require expensive GPU time, are slow at inference, and often degrade non‑target concepts.

Training‑free approaches such as NP, SLD and SuppressEOT run at inference speed but suffer from limited erasure precision (NP), need manual target localization (SuppressEOT), or poorly preserve prior knowledge (SLD), leading to noticeable quality loss.

Core Principle of AdaVD: Orthogonal Complement Projection

AdaVD is a training‑free method that operates on the value space of the UNet within a diffusion model. For each token in the text prompt, the method projects its value representation v onto the orthogonal complement of the subspace spanned by the target concept token c:

v_{erased}=v - \frac{\langle v, c \rangle}{\|c\|^2}\,c

This orthogonal complement operation removes the component of v that aligns with the target concept, effectively stripping the target semantics while leaving other dimensions untouched. The token‑wise formulation provides fine‑grained control over each word’s contribution.

Adaptive Token Shift Mechanism

To avoid over‑erasing tokens that are only weakly related to the target, AdaVD computes a similarity‑based shift factor \alpha_i for each token i: \alpha_i = \sigma(\text{sim}(t_i, t_{target})) where \sigma is a sigmoid and \text{sim} is cosine similarity in the text embedding space. The final erased value becomes

v_i' = v_i - \alpha_i \cdot \frac{\langle v_i, c \rangle}{\|c\|^2} c

. Tokens with low similarity receive a small \alpha_i, preserving prior knowledge.

Experimental Validation

AdaVD was evaluated on several concept‑erasure benchmarks:

IP characters (e.g., Snoopy, Mickey, SpongeBob)

Artistic styles

NSFW content

Celebrity attributes

Metrics:

CLIP Score (CS) – measures how well the target concept is removed.

FID – measures preservation of overall image quality and non‑target semantics.

Across all tasks AdaVD achieved higher CS (better erasure) and lower FID (better prior preservation) than NP, SLD, SuppressEOT and training‑based baselines. In multi‑concept experiments (simultaneous removal of three characters and even 40 concepts) AdaVD maintained strong CS while keeping FID comparable to the best baselines, demonstrating scalability.

Efficiency and Interpretability

Because no model fine‑tuning is required, AdaVD’s inference time is comparable to other training‑free methods and an order of magnitude faster than training‑based approaches. Visualizations of the erased value maps show that semantic regions associated with the target concept are precisely nulled, while unrelated regions become featureless noise, confirming the method’s balance between erasure and prior protection.

Broader Applications

Beyond explicit concept removal, AdaVD can be used for:

Implicit concept suppression (e.g., removing “rainy” or “foggy” ambience).

Image editing (e.g., deleting glasses, mustaches).

Attribute suppression (e.g., removing red color from apples or roses).

The technique is compatible with a range of diffusion models, including Chilloutmix, DreamShaper, RealisticVision, and Stable Diffusion v2.1.

Resources

Paper: https://arxiv.org/pdf/2412.06143 Code:

https://github.com/WYuan1001/AdaVD