Essential Machine Learning Algorithms Every Beginner Must Know

This article is aimed at machine learning beginners and introduces common machine learning algorithms, welcoming peer discussion.

Philosophy seeks answers to where we come from, who we are, and where we go; similarly, machine learning follows a pipeline: organize data → mine knowledge → predict the future. Organizing data corresponds to feature design, generating samples; mining knowledge is modeling; prediction is applying the model.

Feature design depends on understanding the business scenario and can be continuous, discrete, or high‑order combinatorial features. This article focuses on machine learning algorithms, which can be divided into supervised and unsupervised learning.

Unsupervised learning includes many algorithms; recently, topic models such as LSA, PLSA, and LDA have attracted attention. They differ in modeling assumptions: LSA assumes each document has a single topic, PLSA assumes fixed topic probability distributions, while LDA allows topic probabilities to vary per document and word.

The essence of LDA can be understood via a dice‑throwing analogy; see Rickjin's article "LDA Data Gossip" for an accessible explanation that also covers many mathematical concepts.

Supervised learning splits into classification and regression; the perceptron is the simplest linear classifier, rarely used today but serves as the basic unit of neural networks and deep learning.

Linear functions fitted to data and thresholded for classification are sensitive to noisy samples. Logistic Regression uses a sigmoid to bound outputs between 0 and 1, mitigating noise impact, and is widely used for predicting ad click‑through rates.

Logistic regression parameters are estimated via maximum likelihood: define the likelihood L(θ), take the log to convert products to sums (maximizing likelihood becomes minimizing loss), then solve using gradient descent.

Compared with linear classifiers, nonlinear classifiers like decision trees have stronger classification power; ID3 and C4.5 are typical decision tree algorithms with similar modeling processes, differing mainly in the definition of the gain function.

Linear regression and linear classification share similar forms; the key difference is that classification targets discrete values while regression targets continuous values. Consequently, regression typically uses a least‑squares loss, which under Gaussian error assumptions is equivalent to maximum likelihood.

When solving model parameters with gradient descent, one can use batch or stochastic modes; batch generally yields higher accuracy, while stochastic has lower computational complexity.

As mentioned, the perceptron, though a simple linear classifier, can be regarded as the basic unit of deep learning, with its parameters solvable via methods such as autoencoders.

One advantage of deep learning is feature abstraction: learning high‑level features from low‑level inputs, capturing complex structures. For example, pixel‑level features can be abstracted into edge contours describing texture, which can be further learned into higher‑order representations of object parts.

As the saying goes, three mediocre craftsmen can surpass a genius; whether linear classifiers or deep learning, single models work alone. Model ensemble combines strengths of multiple models to improve accuracy. Bagging, for example, trains multiple models with different algorithms, parameters, or features and aggregates predictions via voting or weighted averaging.

Boosting is another ensemble technique that iteratively adjusts loss weights of misclassified samples to improve overall accuracy; typical algorithms include AdaBoost and GBDT.

Different tasks may choose different ensemble methods; for deep learning, dropout on hidden units achieves a similar effect.

After covering many basic ML algorithms, we discuss evaluation criteria. Underfitting and overfitting are common; a simple check compares training and test errors. To address underfitting, add features; to address overfitting, reduce feature quantity or model complexity.

Feature count reflects model complexity; one can set the number of input features before training, or more commonly add regularization terms on feature parameters to the loss function, allowing the training process to select valuable features.

Model tuning is a meticulous task, ultimately needing to provide reliable predictions for real scenarios. Hope you can apply what you learn!