Understanding Activation Functions in Artificial Neural Networks

Overview

Artificial neural network (ANN) is an artificially constructed neural network based on human understanding of brain neural networks, capable of performing specific functions. It has been widely applied in pattern recognition, prediction, and control systems, and can solve problems that are difficult for conventional computation.

An artificial neuron is the basic building block of an ANN, and the connection weight determines the influence between neurons, so the network input is represented as a vector.

Activation functions, also called excitation or activation functions, are used to transform the network input received by neurons. Generally there are four types:

(1) Linear function.

(2) Nonlinear ramp function, where a constant serves as the saturation value and defines the neuron’s maximum output.

(3) Threshold/step function, where the parameters are non‑negative real numbers and the threshold is a constant. Threshold functions have two special forms: binary and bipolar.

(4) Sigmoid function, where a constant determines the shape; its saturation values are typically 0 and 1, and its simplest form yields these limits.

Reference

司守奎，孙玺菁. Python数学实验与建模.