Understanding Nonlinearity in Machine Learning: From Logistic Regression to Neural Networks

What is nonlinearity?

Nonlinearity describes the gap between raw inputs and target decisions that cannot be captured by a simple linear mapping. When the relationship between variables depends on context, a linear model fails to separate the classes.

Example: height‑weight classification

A logistic‑regression task is to classify body type ("overweight" vs. "underweight") using only height and weight. The same weight can correspond to very different body types depending on height, so a linear decision boundary is insufficient.

Feature engineering to reduce nonlinearity

Include the remainder of height modulo 2 (X%2) as an additional feature, exposing a simple nonlinear pattern.

Convert raw measurements to a binary sequence and let the model use the least‑significant bit as a direct indicator.

Derive Body‑Mass‑Index (BMI) as BMI = weight / height², which captures the relationship more linearly.

Model‑level enhancements

A kernel‑augmented logistic regression can capture interactions, for example: sigmoid(ax + by + k·x·y⁻² + c) Using a periodic activation function also introduces a smooth, differentiable nonlinearity:

y = 0.5 * cos(π * (x - 1)) + 0.5

Speech processing as a high‑nonlinearity domain

Time‑frequency transforms (e.g., spectrograms) and MFCC features are classic examples of engineering nonlinearity to make raw audio amenable to learning.

Limitations and trade‑offs

Adding polynomial or kernel terms to linear models can cause multicollinearity, making weight estimates unstable and violating assumptions of related models such as Naïve Bayes.

Noisy or irrelevant engineered features may lead to over‑fitting and degrade generalization.

Nonlinear extensions increase computational cost.

Effective machine‑learning solutions balance the expressive power gained from nonlinearity with these risks.