This article walks through implementing a basic multi‑layer neural network using only NumPy, covering terminology, network architecture, forward and backward propagation, activation functions, loss calculation, parameter updates with SGD, and compares the custom model with a Keras implementation.
This article explains the fundamental gradient‑descent optimizers used in neural networks—SGD, Momentum, RMSProp, Adam and their variants—illustrates loss‑surface challenges such as local minima, saddle points and ravines, and shows how techniques like mini‑batching, momentum, adaptive learning rates and scheduling address these issues.
This article explains the backpropagation training algorithm for fully connected artificial neural networks, detailing its gradient‑descent basis, mathematical derivation, matrix formulation, and provides a complete Python implementation with mini‑batch stochastic gradient descent, momentum, learning‑rate decay, and experimental results.
This article explains why vanilla Stochastic Gradient Descent often struggles in deep learning, describes the challenges of valleys and saddle points, and introduces three major SGD variants—Momentum, AdaGrad, and Adam—detailing their motivations, update rules, and advantages.
This article explains how Facebook tackles massive recommendation data—over 100 billion ratings—by using distributed collaborative filtering, matrix factorization, SGD/ALS hybrid algorithms, and a novel work‑to‑work communication scheme built on Apache Giraph to achieve high performance and scalability.
The article explains how Facebook built a large‑scale distributed recommendation system using Apache Giraph, collaborative filtering with matrix factorization, SGD and ALS algorithms, a novel work‑to‑work communication scheme, and performance optimizations that achieve ten‑fold speedups on billions of ratings.
