How Depth-Guided Texture Diffusion Boosts Image Semantic Segmentation

Paper Information

Title: Depth-Guided Texture Diffusion for Image Semantic Segmentation

Authors: Wei Sun, Yuan Li, Qixiang Ye, Jianbin Jiao, Yanzhao Zhou

Code repository: https://github.com/Wistzz/Texture-Diffusion.git

Key Contributions

Depth‑guided texture diffusion that extracts high‑frequency texture and edge cues from RGB images and injects them into depth maps, narrowing the modality gap.

Structural consistency optimization using an SSIM‑based loss to keep the enhanced depth map structurally aligned with the RGB image.

Four modular components – Texture Extraction (TE), Texture Diffusion (TXD), Joint Embedding (JEB) and Adapter – each validated by ablation studies.

Method Overview

The framework consists of three stages: texture extraction, texture diffusion, and depth integration into a segmentation backbone. For camouflaged object detection (COD) and salient object detection (SOD) the backbone is HitNet; for indoor semantic segmentation the backbone is DFormer.

1. Texture Extraction (TE)

Input RGB images are down‑sampled and transformed to the frequency domain with a Fast Fourier Transform (FFT). A high‑pass filter retains only high‑frequency components, which correspond to fine texture and edge details. The filtered spectrum is inverse‑FFT‑ed back to the spatial domain, producing a texture image that preserves local detail while remaining translation‑invariant.

2. Texture Diffusion (TXD)

The extracted texture image is fused into the depth map via a message‑passing process in a latent feature space. Convolutional blocks first embed the raw depth map into a set of latent channels (e.g., 24 channels). For each channel, a diffusion weight map is predicted from the texture cues using a small convolutional network; the weight map defines a spatially varying kernel (e.g., 7×7) that is applied to the latent features. The diffusion is iterated several times (typically 3–5 iterations), each step updating the latent representation:

h^{(t+1)} = h^{(t)} + \text{Conv}(h^{(t)}; \theta_{tex}) \odot w_{tex}

where h^{(t)} is the latent depth feature at iteration t, \theta_{tex} are convolution parameters, and w_{tex} are texture‑derived diffusion weights. After diffusion, the latent features are decoded back to a depth map that now carries texture details.

3. Structural Consistency Optimization

To prevent the diffusion from distorting the geometric structure of the depth map, a structural consistency loss is added. The loss measures the structural similarity (SSIM) between the enhanced depth map D_{enh} and the original RGB image I_{rgb}:

\mathcal{L}_{SC} = 1 - \text{SSIM}(D_{enh}, I_{rgb})

The total training objective combines the segmentation loss \mathcal{L}_{seg} with the weighted structural loss:

\mathcal{L}_{total} = \mathcal{L}_{seg} + \lambda_{SC}\,\mathcal{L}_{SC}

where \lambda_{SC} balances the influence of structural consistency.

4. Depth Integration in the Segmentation Network

Joint Embedding (JEB) : The enhanced depth map is up‑sampled to the RGB resolution, replicated to three channels, and added element‑wise to the RGB feature map. The summed representation is processed by a ConvNeXt‑style embedding block to produce a fused feature tensor.

Adapter Modules : At each stage of the backbone (HitNet or DFormer) an Adapter consisting of a 1×1 convolution followed by ReLU aligns the channel dimensions of the depth features with the backbone’s feature maps. The adapted depth features are then added to the backbone activations, preserving the original architecture while enriching it with depth cues.

Experiments

Extensive evaluation on camouflage object detection (COD), salient object detection (SOD) and indoor semantic segmentation benchmarks shows consistent improvements over state‑of‑the‑art baselines. The method works with both estimated depth (e.g., from monocular depth predictors) and sensor‑captured depth. Quantitative gains include higher mean F‑measure and mean Intersection‑over‑Union (mIoU) across all datasets. Ablation studies confirm:

Removing TE degrades performance, indicating the importance of high‑frequency texture cues.

Omitting the structural consistency loss leads to noticeable drops in boundary accuracy.

Adapter modules contribute additional gains without increasing backbone parameters.

Conclusion

The depth‑guided texture diffusion framework effectively bridges the modality gap between depth and RGB by injecting high‑frequency texture cues into depth maps and enforcing structural alignment via SSIM. The modular design (TE, TXD, JEB, Adapter) can be attached to various segmentation backbones, delivering consistent accuracy gains on diverse segmentation tasks.