Optimizers and Schedulers in Neural Network Architecture: A Detailed Guide

This article introduces the use of optimizers and schedulers to improve model training and hyper‑parameter tuning.

Optimizer basics

Optimizers are a crucial component of neural‑network architectures, while schedulers are essential tools in deep learning. When defining a model, important choices include data preparation, architecture, loss function, and then the optimizer and scheduler. Many projects default to SGD or Adam, but more effective training strategies exist.

Optimizers are defined by three aspects:

1. Optimization algorithm (e.g., SGD, RMSProp, Adam…)

2. Optimization hyper‑parameters (learning rate, momentum…)

3. Training parameters (the set of model parameters to update)

The article focuses on how to leverage items 2 and 3.

Training loop overview

1. Forward pass: compute outputs from current parameters and inputs.

2. Compute loss between outputs and targets.

3. Back‑propagation: calculate gradients of loss w.r.t. parameters.

4. Parameter update: use gradients to adjust parameters for the next iteration.

Model parameters reside in each layer (weights, biases) and are represented as tensors with associated gradients. Frameworks such as PyTorch and Keras provide specific data types for these parameters.

When constructing an optimizer, the user specifies which parameters to update—typically all model parameters. In GANs, two optimizers each manage half of the model’s parameters.

During each update the optimizer uses:

1. Current parameter value

2. Parameter gradient

3. Learning rate and other hyper‑parameter values

For example, the SGD update formula is shown in the following diagram.

Optimizers do not compute gradients for other models.

Hyper‑parameter control

All optimizers require hyper‑parameters; specific algorithms may need momentum, beta, weight decay, etc. The chosen values significantly affect training speed and model performance, so fine‑grained control is important.

Two axes of control are possible. The first involves parameter groups.

Parameter groups

Some networks use a single hyper‑parameter set, but different layers may need distinct settings. By defining multiple parameter groups—each containing a subset of layers—different hyper‑parameters can be assigned, a technique called differential learning.

A common use case is transfer learning. A pretrained model (e.g., ImageNet‑trained CNN) is split into two groups: the feature‑extracting convolutional layers and the final linear classifier layers. The former receives a very low learning rate to preserve learned features, while the latter uses a higher learning rate to adapt to the new task.

PyTorch offers built‑in support for parameter groups, allowing per‑group hyper‑parameter customization. Keras lacks native group support, requiring a custom training loop to partition parameters and apply different hyper‑parameters.

Schedulers

After covering hyper‑parameters, the article turns to schedulers, which modify hyper‑parameters over training epochs—known as adaptive learning rates. Various mathematical curves (exponential, cosine, cyclic) are implemented as built‑in schedulers in PyTorch and Keras. Users select an algorithm, define minimum and maximum values, and the scheduler computes the hyper‑parameter value at the start of each epoch.

The scheduler is an optional, separate component; without it, hyper‑parameters remain constant throughout training. It works alongside, not within, the optimizer.

In summary, the article explains the roles and capabilities of optimizers and schedulers, showing how they can be combined—through parameter groups and adaptive schedules—to achieve finer‑grained control over learning rates and other hyper‑parameters, ultimately enhancing model performance. Both PyTorch and Keras provide built‑in functions to support these techniques.