Mastering Convolutional Neural Networks: Theory, Training, and Implementation

Introduction

Fully‑connected networks are inefficient for image and speech tasks. Convolutional neural networks (CNNs) address these limitations and are the dominant architecture for visual and auditory recognition.

ReLU Activation

Modern CNNs use the Rectified Linear Unit (ReLU) defined as f(x)=max(0,x). ReLU provides sparsity, mitigates vanishing gradients, and is computationally cheap.

Why Convolutional Networks

Fully‑connected nets suffer from:

Parameter explosion : a 1000×1000 input with 100 hidden units requires ~100 M parameters.

Loss of spatial locality : each neuron sees all pixels, ignoring local structure.

Depth limitation : gradients vanish after a few layers, preventing deep models.

CNNs solve these with three principles:

Local connections : each neuron connects to a small spatial region.

Weight sharing : the same filter is applied across the whole image.

Pooling (down‑sampling) : reduces spatial resolution while preserving salient features.

CNN Architecture

A typical CNN repeats the block Convolution → (optional Pooling) followed by one or more fully‑connected layers. The generic pattern can be expressed as: INPUT → [[CONV]*N → POOL?]*M → [FC]*K For example, a network with two convolution‑pooling pairs and two fully‑connected layers follows: INPUT → CONV → POOL → CONV → POOL → FC → FC Here N=1, M=2, K=2.

Three‑Dimensional Layer Structure

Each CNN layer is a 3‑D tensor (width × height × depth). Depth corresponds to the number of feature maps produced by different filters.

Convolution Layer Computation

Given an input X_{i,j}, a filter W_{m,n} with bias W_b, the pre‑activation at position (i,j) is the sum of element‑wise products plus bias. After applying ReLU, the output is: a_{i,j}=f\Big(\sum_{m,n}W_{m,n}\,X_{i+m,j+n}+W_b\Big) The output spatial size for an input of width W1, filter size F, padding P, and stride S is: W2 = floor((W1 - F + 2P) / S) + 1 Similarly for height H2. For multi‑channel input with depth D and N filters, the parameter count is (F·F·D + 1)·N.

Pooling Layer

Pooling reduces spatial dimensions. Common types:

Max pooling : selects the maximum value in each n×n window.

Mean pooling : computes the average of each window.

Depth is unchanged.

Fully‑Connected Layer

The fully‑connected layer performs the standard affine transformation followed by an activation, identical to that described for fully‑connected networks.

Training the CNN

Training uses back‑propagation. The error is propagated:

Through pooling (max pooling routes the error to the maximal element; mean pooling distributes it evenly).

Through convolution, taking stride, padding, and depth into account.

Gradients for filter weights are computed as the cross‑correlation between the upstream sensitivity map and the input.

Bias gradients are the sum of the sensitivity map.

Formulas for stride = 1 are presented; extensions to arbitrary stride S follow the same pattern.

Python Implementation (NumPy)

A minimal implementation includes: import numpy as np A ConvLayer class with __init__, forward, backward, and update_params methods.

Utility functions: calculate_output_size, padding, conv, and element‑wise operations.

Gradient‑checking code to verify the backward pass.

Application: MNIST Handwritten Digit Recognition

The classic LeNet‑5 architecture (two convolution‑pooling pairs followed by two fully‑connected layers) achieves ~0.8 % error on the MNIST test set.

Conclusion

The article covers CNN fundamentals, mathematical formulation, training algorithms, and a hands‑on NumPy implementation, providing a foundation for further topics such as recurrent neural networks.