Essential Machine Learning Algorithms Every Beginner Must Know

This article is aimed at machine learning beginners and introduces common machine learning algorithms; of course, peer discussion is welcome.

Philosophy seeks to answer fundamental questions of origin, identity, and destiny; this quest can be likened to the machine learning workflow: organize data → discover knowledge → predict the future. Organizing data corresponds to feature engineering, generating samples in a required format; discovering knowledge is modeling; predicting the future is applying the model.

Feature design depends on understanding the business scenario and can be categorized into continuous, discrete, and high-order combinatorial features. This article focuses on introducing machine learning algorithms, which can be divided into supervised and unsupervised learning.

There are many unsupervised learning algorithms; in recent years the industry has focused on topic models. LSA → PLSA → LDA represent three typical development stages, differing mainly in their modeling assumptions. LSA assumes each document has a single topic, PLSA assumes fixed topic probability distributions, while LDA assumes the topic probabilities for each document and word can vary.

The essence of the LDA algorithm can be understood by analogy to a god rolling dice; detailed explanations are available in Rickjin's article “LDA Data Gossip”, which is accessible and also introduces many mathematical concepts—highly recommended.

Supervised learning can be divided into classification and regression; the perceptron is the simplest linear classifier, now rarely used in practice, but it serves as the fundamental unit of neural networks and deep learning.

When a linear function fits data and classifies based on a threshold, it is easily affected by noisy samples, reducing accuracy. Logistic Regression uses the sigmoid function to constrain outputs between 0 and 1, effectively mitigating the impact of noise, and is widely used for predicting online ad click-through rates.

Logistic regression model parameters can be solved via maximum likelihood: first define the objective function L(θ), then apply a logarithmic transformation to convert the product into a sum (maximizing likelihood becomes minimizing loss), and finally use gradient descent to solve.

Compared with linear classifiers, nonlinear classifiers such as decision trees have stronger classification power; ID3 and C4.5 are typical decision tree algorithms with similar modeling processes, differing mainly in the definition of the gain (objective) function.

Linear regression and linear classification share similar forms; the essential difference is that classification targets discrete values while regression targets continuous values. This leads regression to typically define its objective via least squares, which under Gaussian error assumptions is equivalent to maximum likelihood.

When solving model parameters with gradient descent, one can use batch or stochastic modes; generally, batch mode yields higher accuracy while stochastic mode has lower computational complexity.

As mentioned earlier, the perceptron, though the simplest linear classifier, can be regarded as the basic unit of deep learning, and its parameters can be solved using methods such as autoencoders.

One advantage of deep learning is feature abstraction: learning high-order features from low-level inputs to describe more complex structures. For example, learning edge and texture descriptors from pixel-level features, then further learning higher-order representations of object parts.

As the saying goes, three mediocre craftsmen can surpass a genius; whether linear classifiers or deep learning, each model works alone. Is there a way to combine many strengths to further improve accuracy? Model ensemble addresses this. Bagging is one method: for a given task, train multiple models with different algorithms, parameters, or features, then combine their predictions via voting or weighted averaging.

Boosting is another ensemble approach; it iteratively adjusts the loss weights of misclassified samples to improve overall accuracy, with typical algorithms such as AdaBoost and GBDT.

Having introduced many basic machine learning algorithms, let's discuss fundamental criteria for evaluating models. Underfitting and overfitting are common; a simple assessment compares training and test errors. When underfitting, design more features to improve training accuracy; when overfitting, reduce feature quantity or model complexity to improve test accuracy.

Feature count directly reflects model complexity; setting the number of input features before training is one approach. Another common method introduces regularization terms on feature parameters into the objective/loss function, allowing the training process to select high-quality features.

Model tuning is a meticulous task; ultimately it must deliver reliable predictions for real-world scenarios and solve practical problems. Hope you can apply what you learn!