Can Diffusion Models Revolutionize Salient Object Detection?

Introduction

Salient Object Detection (SOD) is a crucial task in image processing that mimics human visual attention to quickly extract the most attractive or important regions from an image.

Background

Identifying salient regions enables applications such as avatar creation, ID photos, and creative image synthesis. In medical imaging, SOD helps locate lesions in X‑ray, CT, or MRI scans. In sports video analysis, it can highlight players or the ball to assist coaches in real‑time decision making.

Challenges

Early handcrafted feature methods rely on low‑level cues such as color inconsistency and edge variation, which struggle when salient objects blend with complex backgrounds.

Deep‑learning approaches improve performance via dense feature interaction and attention mechanisms, yet still face over‑confident inaccurate predictions and poor cross‑domain generalization in complex scenes.

A new paradigm and techniques are urgently needed to address these SOD issues.

Motivation

Diffusion models have become powerful generative paradigms capable of learning complex data distributions and producing fine‑grained image edges. By framing SOD as a denoising task, we can leverage diffusion models to generate high‑quality masks.

In the forward process, the salient mask is progressively noised; in the reverse process, the noise is iteratively removed to recover the original mask.

Model Structure

Diffusion Model

The diffusion model originates from a 2015 ICML paper on nonequilibrium thermodynamics and gained popularity after the 2020 DDPM paper at NeurIPS.

Components

Forward diffusion process: Add Gaussian noise to the input image repeatedly.

Reverse diffusion process: Iteratively denoise to recover the original image.

For SOD‑diffusion, the forward step adds noise to the ground‑truth mask, and the reverse step denoises it back.

Innovation

Shift from discriminative to generative SOD: the model generates salient regions by denoising random noise.

Salient encoder SwinB and feature‑fusion module CBAM capture object location and edge details.

Attention Feature Interaction Module (AFIM) couples image features with diffusion features to guide the denoising process.

Model Training

We employ a pretrained Stable Diffusion VAE to encode the image x and salient mask y into latent space, add random noise to y, and use the Attention Feature Interaction Module to implicitly guide the diffusion process, strengthening the coupling between SwinB‑extracted image features and diffusion features. The model is optimized by minimizing diffusion loss as described in the paper “SOD‑diffusion: Salient Object Detection via Diffusion‑Based Image Generators”.

Model Inference

Given an input image x, the Stable Diffusion VAE encodes it to a latent zₓ. This latent is concatenated with the noisy salient latent z_y and fed to the U‑Net at each denoising step. After t steps, the denoised latent z₀ᵧ is decoded to produce the final salient mask, obtained by averaging its three channels.

Experiments

Experimental Setup

We train on DUTS‑TR (10,553 images) using the Stable Diffusion v2 backbone. Evaluation is performed on five unseen benchmarks: DUT‑OMRON, ECSSD, PASCAL‑S, HKU‑IS, and DUTS‑TE, using standard metrics (F‑measure, MAE, E‑measure, S‑measure).

Evaluation Metrics

Overall Results

The first row shows the input image, the second row the ground truth, and the third row the SOD‑diffusion output.

Ablation Study

SwinB effectiveness: Early denoising steps learn partial object locations and edges, refined by auxiliary features during inference.

CBAM effectiveness: Enhances focus on the most salient regions.

AFIM effectiveness: Integrates diffusion and salient features better than vanilla cross‑attention.

Auxiliary loss (AL) effectiveness: Improves overall performance.

Sampling Process Visualization

Conclusion

Limitations

Balancing inference speed with detection accuracy remains challenging.

Future work should explore lightweight architectures, knowledge distillation, and model pruning for real‑time applicability.

Future Work

We propose three key modules—CBAM, AFIM, and auxiliary loss—to compensate diffusion models for SOD.

The generative approach fundamentally changes the SOD paradigm and boosts performance.

Further research will address efficiency and multimodal SOD tasks, establishing a solid foundation for subsequent exploration.

Author Bio

Jiaming Huang, Senior Algorithm Engineer, Image Group, Intelligent Platform Department, Huolala Technology Center.