From Biological Neurons to Artificial Neural Networks: Perceptrons, Multilayer Perceptrons, and Backpropagation

Artificial neural networks (ANNs) are the core of modern deep learning, inspired by the structure and function of biological neurons. Early work by McCulloch and Pitts (1943) introduced a simplified binary neuron model that could compute any logical proposition, laying the foundation for later ANN architectures.

The perceptron, invented by Frank Rosenblatt in 1957, extends the binary neuron to a threshold logic unit (TLU) with weighted inputs and a step activation function. Its learning rule, derived from Hebb’s principle, updates weights proportionally to the error made on each training instance. A concrete Scikit‑Learn example demonstrates how to train a perceptron on the Iris dataset:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import Perceptron
iris = load_iris()
X = iris.data[:, (2, 3)]  # petal length, petal width
y = (iris.target == 0).astype(np.int)  # Iris setosa?
per_clf = Perceptron()
per_clf.fit(X, y)
y_pred = per_clf.predict([[2, 0.5]])

Although a single perceptron can only solve linearly separable problems, stacking multiple perceptrons creates a multilayer perceptron (MLP). An MLP consists of an input layer, one or more fully‑connected hidden layers, and an output layer. Training MLPs became feasible after the 1986 discovery of the back‑propagation algorithm, which efficiently computes gradients of a loss function by a forward pass followed by a reverse pass through the network.

Back‑propagation relies on automatic differentiation (reverse‑mode) to obtain error gradients for all weights, then updates them using gradient descent. Proper random initialization of weights breaks symmetry and enables diverse learning across neurons.

Non‑linear activation functions are essential; without them, a deep stack of linear layers would collapse to a single linear transformation. Common activations include the sigmoid, hyperbolic tangent (tanh), and rectified linear unit (ReLU), each offering different trade‑offs in gradient flow and computational cost.

Overall, the article provides a comprehensive overview of how biological insights led to the development of perceptrons, their limitations, the rise of multilayer networks, and the back‑propagation algorithm that powers today’s deep learning models.