Mastering U‑Net: The Core Engine of Stable Diffusion – Theory to Practice

U‑Net (U‑shaped Network) was proposed in 2015 for medical image segmentation, where limited data and the need for pixel‑level precision demand a network that can extract deep semantic features while preserving spatial detail.

U‑Net Overview

The architecture consists of three main parts: a left‑hand down‑sampling path (Encoder) that repeatedly applies 3×3 convolutions, ReLU, and 2×2 max‑pooling, halving the spatial size and doubling the channel count; a central bottleneck that aggregates the most abstract features; and a right‑hand up‑sampling path (Decoder) that performs up‑convolution, concatenates skip‑connected features from the encoder, and applies two more convolutions to recover resolution. Skip connections copy and crop encoder features so that fine‑grained details survive the up‑sampling process.

Why U‑Net Powers Stable Diffusion

Stable Diffusion (SD) generates images by iteratively denoising a noisy tensor. This requires a model that can predict pixel‑wise refinements at every diffusion step. Because U‑Net already outputs a per‑pixel class map in semantic segmentation, it naturally fits the denoising task. However, the SD version of U‑Net differs from the original in three ways: time‑step embedding, a conditional‑embedding interface, and a modified output layer.

Modifications for Diffusion Models

Time‑step embedding injects the diffusion step into the network.

Conditional embedding provides guidance (e.g., text prompts).

The final convolution is altered to produce the required number of channels for the noise prediction.

Practical Implementation

The following implementation replaces the original encoder with a ResNet‑50 backbone to improve feature extraction.

# Using ResNet50 as encoder
self.resnet = resnet50()  # returns 5 feature maps: feat1~feat5

def forward(self, inputs):
    # Encoder extracts five feature maps
    feat1, feat2, feat3, feat4, feat5 = self.resnet.forward(inputs)
    # Decoder: progressive up‑sampling with skip connections
    up4 = self.up_concat4(feat4, feat5)   # upsample feat5 and concat feat4
    up3 = self.up_concat3(feat3, up4)
    up2 = self.up_concat2(feat2, up3)
    up1 = self.up_concat1(feat1, up2)
    out = self.final_conv(up1)
    return out

Mixed‑precision training is enabled via AMP, and the loss combines optional focal or cross‑entropy loss with Dice loss to handle class imbalance.

# Example loss selection
if focal_loss:
    loss = Focal_Loss(outputs, masks, weights, num_classes=num_classes)
else:
    loss = CE_Loss(outputs, masks, weights, num_classes=num_classes)
if dice_loss:
    loss += Dice_loss(outputs, masks)

The learning‑rate schedule follows a warm‑up cosine decay with a final linear phase.

def get_lr_scheduler(lr_decay_type, lr, min_lr, total_iters,
                     warmup_iters_ratio=0.05, warmup_lr_ratio=0.1):
    def yolox_warm_cos_lr(lr, min_lr, total_iters, warmup_total_iters,
                          warmup_lr_start, no_aug_iter, iters):
        if iters <= warmup_total_iters:
            lr = (lr - warmup_lr_start) * pow(iters / warmup_total_iters, 2) + warmup_lr_start
        elif iters >= total_iters - no_aug_iter:
            lr = min_lr
        else:
            lr = min_lr + 0.5 * (lr - min_lr) * (1.0 + math.cos(math.pi * (iters - warmup_total_iters) / (total_iters - warmup_total_iters - no_aug_iter)))
        return lr
    if lr_decay_type == "cos":
        warmup_total_iters = min(max(warmup_iters_ratio * total_iters, 1), 3)
        warmup_lr_start = max(warmup_lr_ratio * lr, 1e-6)
        no_aug_iter = min(max(no_aug_iter_ratio * total_iters, 1), 15)
        func = partial(yolox_warm_cos_lr, lr, min_lr, total_iters,
                       warmup_total_iters, warmup_lr_start, no_aug_iter)
    return func

Dataset

VOCdevkit/VOC2012/
├── JPEGImages/          # RGB images (.jpg)
├── SegmentationClass/   # pixel‑level masks (0‑20)
├── SegmentationObject/  # instance masks
└── ImageSets/Segmentation/
    ├── train.txt
    ├── val.txt
    └── trainval.txt

Training and Results

The model was trained for 50 epochs (≈4 h). Training loss decreased from ~0.9 to ~0.7, and validation loss stabilised around 0.9 without the typical over‑fitting spike, indicating good generalisation.

Pixel accuracy (PixelAcc): 0.92

Mean IoU: 0.64 (Frequency‑weighted IoU 0.89)

Mean accuracy: 0.76‑0.77

These metrics demonstrate that the U‑Net variant can produce high‑quality pixel‑wise predictions, satisfying the precision requirements of the denoising step in Stable Diffusion.

In summary, the article explains U‑Net’s original design, why it is suitable as the core engine of SD, the three diffusion‑specific adaptations, and provides a complete, reproducible PyTorch implementation with empirical results on a standard segmentation benchmark.