1Classification vs Regression 2Loss Functions 3Gradient Descent 4Feature Scaling 5Hyperparameters 6Epochs & Batching

Lensa ML

Feature Scaling

Step 1 of 6: The Scale Problem

Why different feature ranges cause trouble

2D scatter: equal-scale axes (1 unit = same length on both axes)

Scale factor for Feature 21x

F1 range

18-76

F2 range

35-196

Scale ratio

2.8x

Both features have similar ranges, so the points spread across the plot. Each feature contributes comparably to distances and gradient updates.

Drag Scale factor for Feature 2 from 1x to 100x. Take two people who differ by 20 years of age and 10 units of salary. At 1x their distance is √(20² + 10²) ≈ 22.4, and age supplies 80% of the squared distance. At 100x the salary gap becomes 1,000, and the distance is about 1,000.2, so age barely registers. Gradients scale the same way: in a linear model, the gradient for weight wⱼ is the error times xⱼ, so a feature with large values gets large updates.

Step 2 of 6: Elongated Contours

How unbalanced scales distort gradient descent

Loss contours and gradient descent path:

Scale ratio between features1.0

Scale ratio

1.0x

GD steps

Efficiency

100%

With balanced scales, the loss contours are nearly circular, and gradient descent heads almost straight to the minimum.

Drag Scale ratio between features from 1 to 20. The loss here is ½(w1² + r²w2²), so the curvature is 1 along w1 and r² along w2. That ratio, r², is the condition number. To keep the steep direction stable, the step size here is set to 1.6/r², so the flat direction moves very slowly. At ratio 1 the path takes 22 steps with 100% efficiency. At ratio 4 it takes 31 steps at 34%, and from ratio 8 up it runs into the 80-step cap.

Step 3 of 6: Min-Max Normalization

Squeezing features into [0, 1]

x_norm = (x - min) / (max - min)

Raw data (shared scale) vs. Min-Max normalized [0, 1]:

Select data pointPoint 1

Raw age

34.0

Norm age

0.276

Norm salary

0.000

Min-max normalization squeezes every feature into [0, 1]. On the left, salary (300–2000) dominates age (18–80). On the right, both axes span [0, 1] and the spread in age is visible again.

Move Select data point through the 20 points. Min-max normalization computes x′ = (x − min) / (max − min). Here age runs from about 18.2 to 75.7, so Point 1, aged 34.0, maps to 0.275, and its salary of 353, the lowest in the set, maps to 0. The min and max must come from the training data, so new values can land outside [0, 1]. One extreme outlier squeezes every other value into a small part of the range. Dividing image pixels by 255 is min-max scaling.

Step 4 of 6: Standardization (Z-score)

Centering at zero with unit variance

z = (x - μ) / σ

Raw data (shared scale) vs. Standardized (mean=0, std=1):

Select data pointPoint 1

Raw age

34.0

Z (age)

-0.515

Z (salary)

-1.251

Standardization uses the mean and standard deviation instead of the min and max. On the right, both features are centered at zero with unit variance.

Move Select data point and compare the raw and standardized plots. Standardization computes z = (x − μ) / σ. For these ages the mean is 42.6 and the standard deviation 16.7, so Point 1, aged 34.0, gets z ≈ −0.52. Its salary of 353 gets z ≈ −1.25, since that feature has a mean of 1,065 and a standard deviation of 569. Neither method changes the shape of a distribution. A skewed feature such as income often gets a log transform first, and robust scaling uses the median and interquartile range to resist outliers.

Step 5 of 6: Effect on Training

Scaled features converge faster

Relative loss (normalized to 1.0 at start):

Learning rate0.010

Unscaled

>60

Scaled

Speedup

>1.3x

Both models are stable at this small rate. The unscaled loss falls faster for the first few steps, because half of it sits on the steep axis (eigenvalue 100), which clears quickly. Then it stalls on the flat axis (eigenvalue 1), while the scaled loss keeps falling at the same pace in both directions.

Drag the learning rate from 0.005 to 0.3. The unscaled loss ½(x² + 100y²) is stable only for rates below 2/100 = 0.02. The scaled loss ½(5x² + 5y²) has equal curvature in both directions and stays stable up to 2/5 = 0.4, past the end of the slider. At 0.01 the unscaled curve drops faster for a few steps, then stalls on the flat axis at 15% after 60 steps, while the scaled one reaches 1% in 45. At 0.1 the scaled model converges in 4 steps and the unscaled one diverges.

Step 6 of 6: When to Scale

Which models need it and which do not

Does this model need feature scaling?

Scale-sensitive?

Yes

Type

Gradient-based

When trained with gradient descent, features on different scales cause uneven updates. Unscaled coefficients are also hard to compare. Scale features when you train it with gradient descent or add regularization. A closed-form least-squares fit gives the same predictions either way.

Click Linear Regression, Neural Network, Decision Tree, and k-NN to compare them. A few cases are less obvious. Ordinary least squares solved in closed form gives the same predictions with or without scaling, but ridge and lasso do not, because their penalties depend on the size of each coefficient. PCA needs scaling because it looks for the directions of largest variance. RBF-kernel support vector machines use distances, so they need it too. Tree ensembles, like single trees, split one feature at a time and ignore scale.

What is feature scaling in machine learning?

Feature scaling transforms numeric input features so they share a similar range. The two most common methods are min-max normalization, which maps each feature to [0, 1], and standardization, which shifts each feature to mean 0 and standard deviation 1. Scaling changes the units of the features but keeps the order of the values within each one.

Scaling matters for any model that compares features by distance or learns them with gradient descent. Without it, a feature such as income in dollars swamps a feature such as age in years. k-nearest neighbors, k-means, support vector machines, PCA, and neural networks all depend on it. Tree-based models such as random forests and gradient boosting do not.

Key ideas

Feature scaling puts numeric features on comparable ranges.
Min-max normalization maps a feature to [0, 1] using its minimum and maximum.
Standardization gives a feature mean 0 and standard deviation 1.
Unscaled features stretch the loss surface and slow gradient descent.
Distance-based and gradient-based models need scaling; tree-based models do not.
Scaling statistics must be computed on the training set only.

Common questions

Which is better, normalization or standardization?

Standardization is the usual default, especially for linear models, SVMs, and neural networks, and it copes better with outliers. Min-max normalization suits features with known fixed bounds, such as pixel intensities, or models that expect inputs in [0, 1]. Trying both and comparing validation scores settles the question for a given dataset.

Should scaling be done before or after splitting the data?

After. Fit the scaler on the training set only, then apply the same transformation to the validation and test sets. Computing the mean, standard deviation, minimum, or maximum on the full dataset leaks information about the test data into training and inflates test scores.

Does the target variable need to be scaled?

For classification, no. For regression with a neural network, standardizing or log-transforming a target with large values often makes training more stable. Predictions then have to be converted back to the original units with the inverse transformation.

Does batch normalization replace input scaling?

Not entirely. Batch normalization rescales the activations inside a network during training, which reduces sensitivity to scale in later layers. Scaling the raw inputs is still standard practice, because it is cheap and keeps the first layer well conditioned.

Related explainers

Data Preprocessing — Cleaning, scaling, and encoding raw data
Gradient Descent — Optimizing by following the slope downhill
K-Nearest Neighbors — Classification by proximity and distance metrics
Batch Normalization — Why normalizing activations speeds up deep network training
Norms & Distance Metrics — L1, L2, cosine similarity — measuring closeness in vector spaces
Dimensionality Reduction — PCA and t-SNE — seeing high-dimensional data

References

Last updated October 25, 2026