From Functions to Transformers: Mastering Neural Networks Step by Step

From Functions to Neural Networks

Mathematical functions map input data to output data; the classic example is the Pythagorean theorem, which expresses a geometric relationship as a function.

Early AI pursued symbolic approaches, but complex real‑world patterns led to connectionist methods that learn approximate functions from data.

By applying an activation function such as f(x) = (ax + b)^2 , a linear relationship becomes non‑linear, enabling the construction of deep neural networks through successive layers of linear transformations and activations, known as forward propagation .

Computing Neural Network Parameters

Model performance is measured by a loss function, often the sum of absolute errors or the mean‑squared error (MSE). Minimizing loss involves taking partial derivatives with respect to each parameter (weights w and bias b ), a process called gradient descent , where the learning rate controls step size.

The gradient vector points in the direction of steepest increase; moving opposite to it reduces loss, forming one training iteration.

Training Neural Networks

Repeated forward passes compute predictions, while backward propagation (the reverse‑order application of the chain rule) calculates gradients for each layer, completing one training round.

Tuning Neural Networks

Overfitting occurs when a model fits training data perfectly but fails on unseen data; improving generalization involves simplifying the model, adding more data, or using techniques like early stopping.

Regularization adds a penalty term (L1 or L2) to the loss to discourage large weights, with a regularization coefficient acting as a hyper‑parameter.

Dropout randomly disables a subset of neurons during training, reducing reliance on any single feature.

Matrix Operations

Representing computations as matrix multiplications makes formulas concise and leverages GPU parallelism for faster training.

From Word Embedding to RNN

One‑hot encoding maps each token to a high‑dimensional sparse vector, while word embeddings provide dense, lower‑dimensional representations that capture semantic similarity.

Recurrent Neural Networks (RNNs) pass hidden states forward, allowing the model to retain information about previous tokens, but suffer from vanishing gradients and limited parallelism.

Transformer

The Transformer replaces recurrence with multi‑head attention , where queries (Q) compare against keys (K) to weight values (V), enabling each token to attend to all others simultaneously.

Encoder layers apply self‑attention and feed‑forward networks to produce K and V matrices; decoder layers use masked self‑attention (Q) and cross‑attention with encoder outputs to generate predictions, finally projecting to a vocabulary distribution via softmax.

Modern chat models are built on the decoder side of the Transformer, repeatedly predicting the next token.