Spectral Normalization in GANs: Theory, Implementation, and Code Walkthrough

Spectral Normalization in GANs – Theory and Code Analysis

Introduction

The author introduces the series on Generative Adversarial Networks (GANs) and recommends reading previous tutorials on DCGAN and WGAN before diving into Spectral Normalization.

Spectral Normalization Theory

After a brief recap of WGAN, the article highlights the difficulty of satisfying the Lipschitz continuity condition for the discriminator. While WGAN uses weight clipping and WGAN‑GP adds a gradient penalty, Spectral Normalization achieves the same goal by normalizing each weight matrix with its spectral norm.

The Lipschitz condition is expressed as |f(x1) - f(x2)| ≤ K |x1 - x2| , which limits the gradient norm of the discriminator. By ensuring the product of the spectral norms of all weight matrices is ≤ 1, the discriminator becomes 1‑Lipschitz.

To enforce this, each layer’s weight matrix W is divided by its largest singular value σ(W). The article explains how the singular value is the square root of the largest eigenvalue of WᵀW and that diagonal matrices arising from ReLU activations have singular values ≤ 1.

Implementation Details

The provided PyTorch code implements the power‑iteration method to approximate the spectral norm. The key functions are _update_u_v and l2normalize . The algorithm iteratively updates left and right singular vectors u and v , computes σ, and rescales the weight tensor.

def _update_u_v(self):
    u = getattr(self.module, self.name + "_u")
    v = getattr(self.module, self.name + "_v")
    w = getattr(self.module, self.name + "_bar")
    height = w.data.shape[0]
    for _ in range(self.power_iterations):
        u.data = l2normalize(torch.mv(w.view(height, -1).data, v.data))
        v.data = l2normalize(torch.mv(torch.t(w.view(height, -1).data), u.data))
    sigma = u.dot(w.view(height, -1).mv(v))
    setattr(self.module, self.name, w / sigma.expand_as(w))

def l2normalize(v, eps=1e-12):
    return v / (v.norm() + eps)

Lines 6‑10 correspond to the power‑iteration steps that approximate the spectral norm, and line 11 rescales the weight matrix by σ.

Practical Tips

The author notes that the code uses the CIFAR‑10 dataset for a standard GAN image‑generation task and encourages readers to experiment with the implementation, debug the code, and explore further extensions.

Conclusion

Spectral Normalization provides a simple yet effective way to enforce the Lipschitz constraint in GAN discriminators, improving training stability. Understanding both the mathematical derivation and the concrete PyTorch implementation equips practitioners to apply this technique to their own models.

References

GAN — Spectral Normalization Spectral Normalization for GAN Detailed explanation of Spectral Normalization Understanding Spectral Normalization