DC-ControlNet: Decoupling Control Conditions for More Flexible and Precise Image Generation

Overview

The paper presents DC‑ControlNet, a framework that decouples global control conditions into hierarchical element‑wise content and layout signals, allowing users to combine and edit multiple conditions independently for more accurate and flexible image generation.

Method

Preliminaries

Diffusion models (DMs) generate images by iteratively denoising latent variables; the loss is mean‑squared error between predicted and true noise, conditioned on time step and optional control signals.

Decoupled ControlNet

DC‑ControlNet replaces the original ControlNet’s single global condition with two controllers:

Intra‑element controller processes each element’s content condition (e.g., edge, depth, color) together with its layout condition (point, box, or mask) using layout embeddings, residual blocks, and a cross‑attention transformer that injects content features into the UNet.

Inter‑element controller fuses multiple element features, resolves occlusion by assigning a one‑dimensional order embedding, and applies spatial‑ and layer‑wise re‑weighting transformers (Algorithms 1 and 2) to predict per‑pixel and per‑layer weights.

Cross‑normalization restores the original feature distribution after fusion, preventing training instability.

Loss Function

Following prior work, foreground pixels receive higher weight inversely proportional to their area; a feature‑level L1 loss aligns transformed features with target ControlNet features, both weighted by a balancing coefficient.

Experiments

DMC‑120k Dataset

A new 120k‑sample dataset is built by generating multi‑element images with SDXL and FLUX, detecting objects with GroundingDINO, extracting masks, handling occlusions via image inpainting, and providing diverse conditions (Canny, HED, depth, segmentation, normal, point, box, mask).

Setup

All models are trained on eight A100 GPUs with mixed precision. Training proceeds in three stages (union ControlNet, intra‑element controller, inter‑element controller) using AdamW (lr = 1e‑4), 50 k steps, batch size 32, and a 0.2 dropout on prompts.

Results

Qualitative comparisons (Fig 7‑9) show that DC‑ControlNet can independently control element content and layout, resolve overlapping regions by adjusting layer order, and avoid the artifacts seen in ControlNet, UniControlNet, ControlNet++, Layout Diffusion, and HiCo. Quantitative metrics on the DMC‑120k benchmark confirm superior flexibility and precision.

Ablation Study

Removing the order embedding, layer transformer, or spatial transformer degrades performance: without order embedding the model misinterprets element hierarchy; without layer transformer the model cannot distinguish which element should appear in the foreground; without spatial transformer noticeable artifacts appear in overlapping areas (Fig 10).

Conclusion

By decoupling global conditions into element‑wise content and layout signals and introducing dedicated intra‑ and inter‑element controllers, DC‑ControlNet achieves more precise, flexible, and conflict‑aware multi‑condition image generation, outperforming existing ControlNet‑based methods.