Introduction to Artificial Neural Networks and BP Neural Network Implementation with Keras and Scikit-learn

1. Artificial Neural Network

Artificial Neural Network (ANN) is inspired by biology and simulates the brain by using simple computational units as neurons, densely connected to form a network. The structure is illustrated below:

The ANN model combines weighted signals (w1,w2,…,wn) with inputs (x1,x2,…,xn), aggregates them (z), applies an activation function f(x), and outputs the result.

2. Types of Activation Functions

(1) Threshold function:

(2) ReLU function:

(3) Piecewise linear function.

(4) Non‑linear transfer function.

3. Common ANN Models

(1) BP Neural Network (Back‑Propagation).

(2) RBF Neural Network.

(3) FNN.

(4) LM Neural Network (high precision).

4. BP Neural Network

Idea: errors are propagated backward layer by layer from the output to compute hidden‑layer errors.

Model diagram:

Algorithm stages:

First stage (forward): inputs pass through each hidden layer to compute neuron outputs.

Second stage (backward): output error is propagated backward to compute hidden‑layer errors and adjust weights.

Workflow diagram:

Implementation for classification and regression:

Classification – using Keras Sequential module:

Data processing:

Model building:

Training and prediction results are then obtained.

Classification – using scikit‑learn’s MLPClassifier:

class sklearn.neural_network.MLPClassifier(hidden_layer_sizes=(100,), activation='relu', solver='adam', alpha=0.0001, batch_size='auto', learning_rate='constant', learning_rate_init=0.001, power_t=0.5, max_iter=200, random_state=None)

Key parameters:

hidden_layer_sizes: tuple indicating number of neurons per hidden layer.

activation: {'identity','logistic','tanh','relu'} (default 'relu').

solver: {'lbfgs','sgd','adam'} (default 'adam').

Activation functions:

identity – linear (f(x)=x).

logistic – sigmoid (f(x)=1/(1+exp(-x))).

tanh – hyperbolic tangent.

relu – max(0,x).

Solvers:

lbfgs – quasi‑Newton optimizer, fast on small datasets.

sgd – stochastic gradient descent.

adam – adaptive moment estimator, works well on large datasets.

Regression – using scikit‑learn’s MLPRegressor:

sklearn.neural_network.MLPRegressor()

Resulting regression output is shown in the following figure.

Declaration: Content sourced from https://www.jianshu.com/p/4b8e15035826