Linear Regression Basics: Gradient Descent & Feature Standardization with PyTorch

Linear regression is one of the simplest model-building techniques. The article uses Google's car weight vs. miles-per-gallon example: each point represents a sample, and the fitted line becomes a predictive model.

Key Terminology

Feature — input variable (e.g., car weight); can be multiple.

Label — target value to predict (e.g., MPG).

Weight — parameter vector matching feature dimensions.

Bias — scalar offset.

Training adjusts weight and bias so predictions y' approach the label by minimizing the loss function , most commonly Mean Squared Error (MSE) .

Gradient Descent Algorithm

Because solving for zero derivatives analytically is impractical, gradient descent is used. Four steps:

Initialize weight and bias with random values.

Compute gradients — partial derivatives of loss w.r.t. each parameter.

Update parameters — move opposite the gradient scaled by learning rate .

Stop after a fixed number of iterations.

Initial values and learning rate are hyperparameters set by experience.

PyTorch Implementation

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Generate synthetic data
inputs = torch.rand(1024, 2)
weights = torch.tensor([[42.0], [24.0]])
bias = torch.tensor(8.0)
targets = inputs @ weights + bias + 0.1 * torch.randn(1024, 1)

# Initialize learnable parameters
w = torch.rand((2, 1), requires_grad=True, device=device)
b = torch.rand((1,), requires_grad=True, device=device)

inputs = inputs.to(device)
targets = targets.to(device)

epoch = 10000
lr = 0.003

for i in range(epoch):
    outputs = inputs @ w + b
    loss = torch.mean(torch.square(outputs - targets))
    print("loss:", loss.item())
    
    # Compute gradients
    loss.backward()
    
    # Update parameters
    with torch.no_grad():
        w -= lr * w.grad
        b -= lr * b.grad
    
    w.grad.zero_()
    b.grad.zero_()

print("权重 w:", w)
print("偏置 b:", b)

The backward() call computes gradients automatically. With well-scaled features the loss curve drops quickly and converges near zero.

Failure Case: Unscaled Features

Multiplying the second input column by 100: inputs[:, 1] *= 100 causes the loss to diverge instead of converging.

Root Cause Analysis

Both weight parameters share a single learning rate, but their feature ranges differ by two orders of magnitude. This makes gradients for the large-scale feature dominate, preventing effective updates for the small-scale feature.

Solution: Feature Standardization

Normalize each feature to zero mean and unit variance:

# Compute feature mean and std
mean = inputs.mean(dim=0)
std = inputs.std(dim=0)

# Standardize features
inputs_norm = (inputs - mean) / std

After standardization, gradient descent converges smoothly again.