Feature Selection Techniques in Machine Learning: Filters, Wrappers, and Embedded Methods

What is Feature Selection?

Feature selection is a preprocessing technique that identifies and retains the most relevant features for building a machine‑learning model.

For example, in a bank dataset containing age, gender, salary, customer ID, account balance, and branch code, the customer ID contributes almost nothing to predicting customer behavior and should be removed.

Why Feature Selection Matters

Real‑world datasets often contain hundreds or thousands of features. Too many features cause the curse of dimensionality, longer training times, higher over‑fitting risk, reduced model generalization, and increased storage needs. Selecting features lets the model focus on truly informative information.

Types of Feature Selection Techniques

1. Filter Methods

Filter methods evaluate each feature independently of any learning algorithm, so they are fast and have low computational overhead, making them suitable for large datasets. The trade‑off is that they ignore interactions between features.

Correlation‑Based Feature Selection

Measures the linear relationship between an input feature and the target variable.

Chi‑Square Test

Used for categorical features; it measures the dependence between a feature and the target. A higher chi‑square score indicates more discriminative information. Typical applications include customer segmentation, text classification, and medical diagnosis.

Mutual Information

Quantifies how much information one variable provides about another. Unlike correlation, mutual information captures nonlinear relationships and is more suitable for classification tasks.

2. Wrapper Methods

Wrapper methods use a machine‑learning model to evaluate subsets of features by repeatedly training on different combinations, thus accounting for feature interactions. This usually yields better performance but at higher computational cost.

Forward Selection

Starts with no features and iteratively adds the feature that most improves performance until no further gain is observed.

Train a model with each individual feature.

Select the feature that yields the best performance.

Iteratively try adding each remaining feature to the current set.

Repeat until performance stops improving.

Backward Elimination

Starts with all features and iteratively removes the least important feature until the optimal subset remains.

Train a model with all features.

Remove the least important feature.

Retrain the model.

Repeat until the feature set converges.

The resulting subset is usually of high quality and easy to interpret.

Recursive Feature Elimination (RFE)

RFE is a widely used wrapper method with the following workflow:

Train a model.

Rank all features according to their importance.

Remove the lowest‑ranked feature.

Retrain the model.

Repeat until the desired number of features remains.

Example: In a bank dataset with original features age, salary, balance, credit score, and transaction history, RFE selects salary, balance, and credit score as the final features.

3. Embedded Methods

Embedded methods perform feature selection during model training, combining the efficiency of filter methods with the interaction awareness of wrapper methods.

Lasso Regression (L1 Regularization)

Lasso adds an L1 penalty to the loss function, driving coefficients of irrelevant features to zero and thus removing them. Loss = RSS + λ ∑ |β_i| where RSS is the residual sum of squares, λ is the regularization parameter, and β are the coefficients. As λ increases, more coefficients become zero, eliminating the corresponding features.

Tree‑Based Feature Importance

Decision trees, random forests, and XGBoost naturally compute feature importance by aggregating the contribution of each feature at split nodes. Higher importance scores indicate a larger impact on predictions.

Conclusion

Method selection follows practical guidelines: correlation analysis is fast for removing highly collinear features; RFE performs reliably on medium‑sized datasets; Lasso is suited for linear models with many features; tree‑based importance works for both classification and regression tasks. In most cases, combining several methods yields more reliable results than relying on a single technique.

Feature selection is an indispensable step in the machine‑learning pipeline. Removing irrelevant and redundant features improves model performance, computational efficiency, and interpretability. The choice among correlation analysis, chi‑square test, mutual information, RFE, Lasso regression, and tree‑based importance depends on dataset size, feature types, and the algorithm used.