Unveiling the Mathematics Behind Deep Learning Success

In recent years deep learning, particularly convolutional neural networks (CNNs), has achieved great success in image recognition, yet its black‑box nature puzzled theorists. This article aims to reveal the mathematical reasons behind deep learning's success by focusing on three core elements—architecture, regularization, and optimization—and summarizing recent proofs of properties such as global optimality, geometric stability, and invariant representations.

1. Introduction

Deep networks are parameterized models that apply a sequence of operations to input data. Each operation, called a layer, consists of a linear transformation (e.g., a convolution) followed by a pointwise non‑linear activation (e.g., Sigmoid). Their success in speech, NLP, and computer vision stems from having many layers, architectural tweaks such as ReLU and residual shortcuts, massive datasets like ImageNet, and efficient GPU hardware.

The remarkable performance of CNNs has raised theoretical questions. Understanding the three core factors—architecture, regularization, and optimization—is essential for training high‑performing networks and explaining their inevitability.

A. Approximation, Depth, Width, Invariance

Network architecture can approximate any function, but the required depth and width affect capacity. Early work showed single‑hidden‑layer networks with Sigmoid activations are universal approximators. However, wide shallow networks can be replicated by deeper ones, often with better performance, possibly because depth captures data invariances (e.g., object class invariance to viewpoint or illumination). Recent progress on scattering networks demonstrates that structured convolutional filter banks yield stable, locally invariant representations, explaining part of modern CNN generalization.

B. Generalization and Regularization

Traditional statistical learning theory predicts sample complexity grows polynomially with network size, yet deep networks train with far more parameters than data (the N ≫ D regime). Simple regularization techniques such as Dropout prevent over‑fitting by randomly freezing subsets of parameters each iteration. Recent work using compressed sensing and dictionary learning shows that deep networks with random Gaussian weights embed data while approximately preserving distances, offering a metric‑learning perspective on generalization error.

C. Information‑Theoretic Properties

Learning useful data representations can be framed via information theory, complexity, or invariance criteria. The information‑bottleneck loss, a relaxed notion of minimal sufficient statistics, can be expressed as a sum of cross‑entropy terms plus a regularizer (e.g., adaptive dropout). This leads to maximally disentangled representations and improves robustness to adversarial perturbations, suggesting information‑theoretic regularizers play a key role in deep learning.

D. Optimization

Training uses back‑propagation to minimize a regularized loss, typically via stochastic gradient descent (SGD). Although SGD is analytically tractable for convex losses, deep learning loss surfaces are non‑convex, making global optimality guarantees difficult. Empirical evidence shows SGD often finds flat minima with good generalization; recent methods such as Entropy‑SGD exploit this by targeting wide basins, linking to Hamilton‑Jacobi‑Bellman PDEs and proximal optimization techniques.

Research demonstrates that for certain network families the loss and regularizer are positively homogeneous, implying that critical points are either saddle points or plateaus, and that global minima can be reached without encountering poor local minima.

2. Preliminaries

... (section omitted for brevity) ...

3. Global Optimality in Deep Learning

Learning network parameters from N training samples (X, Y) can be expressed as minimizing a regularized empirical loss:

where L is the loss, Θ is a regularizer (e.g., weight decay), and λ > 0 balances them.

A. Non‑convex Challenges

The optimization problem is non‑convex because the network output Φ(X, W) is a non‑linear function of the weights, making standard algorithms only guarantee convergence to critical points (saddles, local minima, etc.).

B. Single‑Hidden‑Layer Optimality

Early results show that with linear activations and a single hidden layer, the squared loss has a global minimum and all other critical points are saddles. With non‑linear activations, counter‑examples exist where back‑propagation fails even on separable data.

C. Random Weights and Inputs

Recent studies using random matrix theory and statistical physics analyze error surfaces, suggesting that high‑dimensional critical points are more likely saddles than local minima.

D. Positively Homogeneous Networks

Deterministic analyses prove that for sufficiently large positively homogeneous networks, only saddles and plateaus matter; there are no spurious local minima.

4. Geometric Stability

Defining the inductive bias of deep models mathematically helps explain their success. In vision, convolutional architectures provide a bias that promotes geometric stability, which is crucial for robust performance.

5. Structure‑Based Theory

A. Data Structure in Networks

Assuming random i.i.d. Gaussian weights, recent work shows that networks preserve the metric structure of data across layers, enabling stable recovery of original features.

B. Generalization Error

The relationship between data structure and learned network error leads to studies of the gap between empirical and expected error, offering insights into why deep networks generalize.

Authors: René Vidal, Joan Bruna, Raja Giryes, Stefano Soatto