Fundamentals of Deep Learning: Neural Networks, CNNs, RNNs, LSTM, and GRU

Deep learning has been a dominant research area since 2006, with its most visible successes in computer vision, speech recognition, and natural language processing. The industry is now extending these techniques to domains such as gaming, recommendation systems, and advertising.

Three categories of deep model architectures are described: forward feedback networks (e.g., MLP, CNN), backward feedback networks (e.g., stacked sparse coding, deconvolutional nets), and bidirectional feedback networks (e.g., deep Boltzmann machines, stacked auto‑encoders).

The basic computational element of an artificial neural network is the neuron (or perceptron), which receives multiple inputs, applies weighted sums and a bias, and passes the result through an activation function such as sigmoid.

Multi‑layer perceptrons (MLP) consist of one or more hidden layers of neurons. The output of an MLP is obtained by propagating inputs forward through these layers.

Training deep neural networks relies heavily on the learning rate. A large learning rate can cause divergence, while a small rate slows convergence and may trap the model in local minima. Common strategies to adjust the learning rate include constant decay, factor decay, and exponential decay.

Gradient descent is the primary optimization algorithm for minimizing loss functions. The basic version (batch gradient descent) updates parameters after processing the entire dataset, whereas stochastic gradient descent (SGD) updates after each mini‑batch, improving speed and generalization.

Momentum accelerates SGD by incorporating a moving average of past gradients, helping the optimizer escape shallow minima. The update rule is v_t = γ·v_{t-1} + η·∇L(θ) and θ = θ - v_t , where γ is the momentum coefficient.

Weight decay (L2 regularization) adds a penalty term λ·||θ||^2 to the loss, discouraging large weights and reducing over‑fitting.

Batch normalization (BN) normalizes mini‑batch inputs to zero mean and unit variance, which speeds up training and improves stability. The BN transformation is ŷ = γ·(x-μ)/σ + β .

Convolutional Neural Networks (CNN) are composed of alternating convolution, pooling, and classification layers. Convolution layers apply learnable kernels to extract spatial features, pooling layers down‑sample feature maps (using max or average pooling), and fully‑connected classification layers map the final feature representation to class scores via softmax.

The number of parameters in a convolution layer is calculated as (F×F×M_{l-1}+1)×M_l , where F is the kernel size, M_{l-1} the number of input feature maps, and M_l the number of output feature maps. Memory requirements follow Mem_l = N_l×N_l×M_l .

Recurrent Neural Networks (RNN) process sequences by feeding the hidden state from the previous time step into the current step. The Elman RNN updates hidden state as h_t = σ_h(W_h x_t + U_h h_{t-1} + b_h) and output as y_t = σ_y(W_y h_t + b_y) . The Jordan variant replaces the recurrent connection with the previous output.

Long Short‑Term Memory (LSTM) networks introduce a cell state and three gates (input, forget, output) to mitigate the vanishing‑gradient problem. The gate equations are:

i_t = σ(W_i x_t + U_i h_{t-1} + b_i) f_t = σ(W_f x_t + U_f h_{t-1} + b_f) o_t = σ(W_o x_t + U_o h_{t-1} + b_o)

Gated Recurrent Units (GRU) simplify LSTM by merging the input and forget gates into an update gate, reducing parameter count while retaining comparable performance. The GRU update equations are:

z_t = σ(W_z x_t + U_z h_{t-1}) r_t = σ(W_r x_t + U_r h_{t-1}) h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ tanh(W_h x_t + U_h (r_t ⊙ h_{t-1}))

The article concludes with references to further reading on LSTM, RNN effectiveness, and comprehensive surveys of deep learning architectures.