How to Verify Gradient Implementations: A Practical Optimization Exercise

Introduction

Optimization is a branch of applied mathematics and a core component of machine learning. A machine‑learning algorithm can be viewed as the combination of model representation, model evaluation, and an optimization algorithm that searches the hypothesis space for the model with the best evaluation metric.

Different models and evaluation criteria require different optimization methods; classic examples are SVM (linear classifier + maximum margin), logistic regression (linear classifier + cross‑entropy), and CART (decision‑tree model + Gini impurity).

With the rapid growth of big data and deep learning, practitioners now face large‑scale, highly non‑convex problems that challenge traditional convex‑optimization theory. Designing efficient and accurate algorithms for these new scenarios has become a major research focus. Although optimization dates back to Lagrange and Euler, most algorithms used to train deep neural networks, such as Adam Adam [2], were introduced only in recent years.

Most machine‑learning libraries already provide common optimizers, allowing a single line of code to invoke them. Nevertheless, understanding the underlying principles remains essential for developing better solutions.

Gradient Verification Exercise

Scenario : Computing the gradient of the objective function is the key step in solving an optimization problem. In many machine‑learning applications, especially deep neural networks, the gradient formula is complex, and we need to verify that our implementation is correct.

Problem : Given an implementation that can compute the objective value and its gradient (illustrated in the figure below), how can we use the objective‑value function to validate the correctness of the gradient implementation?

Prior Knowledge : Linear algebra and calculus.

Solution and Analysis

The standard approach is to compare the analytical gradient with a numerical approximation obtained by finite differences. The following figures illustrate the verification process and typical results.

References

Domingos, Pedro. “A few useful things to know about machine learning.” *Communications of the ACM* 55.10 (2012): 78‑87.

Kingma, Diederik, and Jimmy Ba. “Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).