Unlocking Machine Learning Basics: From Perceptrons to Ensemble Models

Introduction

This article is aimed at machine‑learning beginners and introduces common algorithms while inviting peer discussion.

Philosophy seeks answers to fundamental questions; similarly, machine learning follows a workflow: organize data → extract knowledge → predict the future. Organizing data corresponds to feature design, modeling corresponds to knowledge extraction, and applying the model corresponds to prediction.

Feature Design and Learning Paradigms

Feature design depends on understanding the business scenario and can be continuous, discrete, or high‑order combined features. Machine learning algorithms are divided into supervised and unsupervised learning.

Unsupervised Learning

Topic models have attracted attention in recent years. LSA → PLSA → LDA represent three typical development stages, differing mainly in their modeling assumptions. LSA assumes a single topic per document, PLSA assumes fixed topic distributions, while LDA allows topic probabilities to vary per document and word.

The LDA algorithm can be intuitively understood with a dice‑rolling analogy. For a deeper, accessible explanation, see Rickjin’s article “LDA Data Gossip”.

Supervised Learning

Supervised learning splits into classification and regression. The perceptron is the simplest linear classifier; although rarely used directly today, it is the basic unit of neural networks and deep learning.

Linear functions are sensitive to noisy samples, reducing classification accuracy. Logistic regression applies a sigmoid function to constrain outputs between 0 and 1, mitigating noise impact and is widely used for click‑through‑rate prediction in online advertising.

Logistic‑regression parameters are estimated via maximum likelihood: define the objective L(θ), take the logarithm to convert products into sums (maximizing likelihood → minimizing loss), and solve using gradient descent.

Decision‑tree classifiers such as ID3 and C4.5 provide stronger non‑linear classification ability; they share a similar modeling process but differ in how they define the gain (objective) function.

Linear regression and linear classification share a similar form; the key difference lies in the objective: classification targets discrete values, regression targets continuous values. Regression typically uses least‑squares loss, which under Gaussian error assumptions is equivalent to maximum likelihood.

Gradient descent can be performed in batch mode (higher accuracy) or stochastic mode (lower computational cost).

From Perceptron to Deep Learning

Although the perceptron is simple, it can be viewed as the basic unit of deep learning; its parameters can be learned via methods such as auto‑encoders.

Deep learning’s advantage lies in hierarchical feature abstraction: low‑level pixel features are transformed into edge‑based textures, which are further abstracted into higher‑order representations of objects.

Model Ensembles

Combining multiple models can further improve accuracy. Bagging trains several models with different algorithms, parameters, or feature sets on the same data and aggregates predictions via voting or weighted averaging.

Boosting iteratively adjusts sample‑weight losses to focus on previously mis‑predicted instances, with classic algorithms such as AdaBoost and Gradient‑Boosted Decision Trees (GBDT).

For deep learning, dropout can serve a similar purpose by randomly dropping hidden‑layer nodes during training.

Model Evaluation and Tuning

Model quality is judged by comparing training and test errors. Under‑fitting suggests adding features; over‑fitting calls for reducing model complexity or applying regularization.

Feature quantity reflects model complexity. Regularization terms can be added to the loss function to penalize excessive parameters and promote selection of high‑quality features during training.

Fine‑grained model tuning ultimately aims to deliver reliable predictions for real‑world scenarios.