ENID
Lensa ML

How Feature Selection Works

Step 1 of 6: More Columns Can Hurt

Noise columns raise training R² and lower the real one

0.60.70.80.904812noise columns addedtraining R²5-fold CV R²next month's sales
Noise columns8

Illustrative data: 60 house sales, price driven by size, age and distance to the centre. Every R² is computed live.

Training R²
0.854
Cross-validated R²
0.723
R² on next month's 400 sales
0.815

A least-squares model predicts price from size, age and distance, plus 8 extra columns of random noise. Each noise column lets the model fit a little more of this sample's quirks, so training R² creeps up to 0.854, while cross-validated R² falls from 0.796 to 0.723 and the score on next month's sales falls too. With 60 rows, every useless column costs real accuracy. That is the case for feature selection: drop what doesn't help, so the model can't learn from it.

Drag the number of noise columns and compare the three curves. Training R² can only go up as columns are added, because least squares can always set a useless weight to zero, so it never reveals the problem. Cross-validated R² and the score on new data fall, because each extra weight is fitted partly to this sample's noise. The fewer rows you have per column, the faster this happens, which is why feature selection matters most on small datasets.

Step 2 of 6: Filter Methods

Score each column alone, and see what that misses

listing_idsizesize_m2agedistancenoise_10noise_8noise_12noise_7noise_6noise_11noise_1noise_3noise_9noise_5noise_2noise_4|r| ≥ 0.30|correlation with price|, one column at a time
real signal near-copy of size pure noise leaks the price
Keep if |r| ≥0.30
Columns kept
6
Cross-validated R²
0.944
R² on next month's sales
-0.044

A filter scores each column on its own, here by its correlation with price, and keeps those above a threshold. It's fast and model-free, and it has three blind spots, all visible here. The top bar is listing_id (r = 0.97): the agency numbered listings after they sold, in price order, so it "predicts" price perfectly in this data and is useless on new listings. With it kept, CV says 0.944 but next month gives -0.044. size_m2 passes right next to size: it's the same information twice, and a filter can't tell. And noise_10, pure noise, passes too: with 60 rows, a random column can correlate at 0.4 by chance. Raise the threshold above 0.97 and nothing is left.

Drag the threshold and watch which columns survive. Filters (correlation, mutual information, chi-squared, ANOVA F) rank columns one at a time against the target. They are cheap enough to run on thousands of columns, which makes them a good first pass. But they can't see redundancy (two copies of the same information both score high), interactions (a column that only matters together with another scores low) or leakage, and with few rows some noise columns score as well as real ones.

Step 3 of 6: Forward Selection

Try every column, keep the best, repeat

chosen so far: (nothing)
round 1: CV R² if this column is addedsize0.520size_m20.501age0.219noise_100.133distance0.125noise_80.028noise_120.007noise_2-0.017noise_9-0.024noise_4-0.031
Best candidate this round
size
CV R² gain from adding it
+0.520
Stop when the gain is under 0.01
keep going

Forward selection is a wrapper method: instead of scoring columns alone, it scores the model. Each round it tries adding every remaining column, cross-validates each candidate model, and keeps the best (listing_id is left out here; step 5 deals with it). Round 1 picks size (CV R² 0.520). Step through the rounds. The cost is many model fits (here 405 across 6 rounds), and greedy steps can miss columns that only help together.

Step through the rounds and watch the candidate bars. Each round fits one model per remaining column, so forward selection costs roughly columns × rounds × folds fits, which gets expensive quickly. It sees redundancy because it judges a column by what it adds to the ones already chosen. Backward elimination starts from all columns and removes the weakest each round; recursive feature elimination (RFE) does the same using the model's coefficients instead of refitting for every candidate.

Step 4 of 6: L1 Regularization

One penalty, exact zeros

0.0050.0220.100.452.00penalty λ (log scale) →coefficient (standardised)
real signal near-copy of size pure noise leaks the price
Penalty λ0.10
Columns with a non-zero weight
7 / 16
Kept
size, age, distance, noise_1, noise_4, noise_8, noise_10
CV R² of a refit on those columns
0.795

The lasso adds λ·Σ|w| to the loss: one penalty, the same for every coefficient. Solving it (here by coordinate descent on standardised columns) pushes each weight towards zero, and any weight whose evidence is weaker than λ lands exactly on zero, which removes that column. Drag λ right and watch the lines hit zero one by one. size_m2 is gone: with two near-copies the lasso tends to keep one and drop the other, here size. Some noise survives too (noise_1, noise_4, noise_8, noise_10): the lasso only sees this sample, and a column that correlates by luck looks like evidence. In practice λ is chosen by cross-validation.

Drag λ and watch the coefficient paths. The L1 penalty's corner at zero is what produces exact zeros; the L2 (ridge) penalty shrinks weights but almost never zeroes them. With strongly correlated columns the lasso tends to pick one arbitrarily, and the choice can flip between samples. Elastic net mixes L1 and L2 to keep correlated groups together. The coefficients are only comparable because the columns were standardised first, so the same λ means the same thing for each.

Step 5 of 6: A Feature That's Too Good

Leaks look like the best column you have

this year's 60 salesnext month (first 120)listing_id → pricelisting_id → price
Correlation of listing_id with price (this year)
0.97
Cross-validated R²
0.940
R² on next month's sales
-0.086

The most dangerous column is the one that looks best. listing_id was assigned after each house sold, in price order, so in this year's data (left) it lines up almost perfectly with price, and cross-validation can't catch it: every fold has the same leak. CV reports 0.940. On next month's listings the ids are just numbers (right), and the model, which leaned on them, scores -0.086: worse than predicting the average. Real examples are common: an ID or timestamp that encodes the outcome, a "days in hospital" column for predicting admission, a field filled in after the fact. The test is not statistical: for each column, ask whether you'd actually know it at prediction time.

Switch listing_id on and off and compare the two scores. Leakage through a feature is the hardest kind to catch, because cross-validation shuffles rows that all share the leak and so confirms it. Warning signs are a single column far more predictive than anything else and a validation score that seems too good. The fix is domain knowledge: drop anything that wouldn't be available when the prediction is made, and validate on data from a later period when you can.

Step 6 of 6: Select Inside the Folds

Otherwise pure noise scores well

-1-0.500.5dataset 1dataset 2dataset 3dataset 4dataset 5R² = 0: no better than the mean
Noise columns200
CV R², chosen on all rows
0.26
CV R², chosen inside folds
-0.51
True R² (the target is noise)
≤ 0

Now everything is noise: 40 rows, 200 random columns and a random target, so no model can do better than predicting the mean (R² = 0). Picking the 8 columns most correlated with the target on all 40 rows, then cross-validating, reports an average R² of 0.26. The validation rows already voted on which columns to keep, so each fold is graded on columns chosen partly for fitting it. More columns give more lucky ones to find. The rule from cross-validation applies to every selection method on this page: filters, forward selection and the lasso's λ all have to be chosen inside the folds, or with nested cross-validation.

Switch between selecting on all rows and inside each fold, then drag the number of noise columns. This is selection bias: choosing the best of many candidates using the same data you then score on. It also applies to choosing λ, the number of features or any other setting by looking at CV scores, which is why a final, untouched test set or nested cross-validation is needed to report an honest number.

What is feature selection in machine learning?

Feature selection is the process of choosing a subset of the input variables in a dataset and discarding the rest before or during model training. Unlike dimensionality reduction methods such as PCA, it keeps the original columns, so the model still works with named, interpretable inputs. The three main families are filter methods, wrapper methods, and embedded methods.

It is used wherever there are many candidate inputs and limited data: genomics studies with thousands of genes per patient, credit models built from hundreds of account variables, and sensor or log data with many redundant signals. Removing irrelevant columns reduces overfitting, speeds up training and inference, cuts the cost of collecting data in production, and makes a model easier to explain to users and regulators.

Key ideas

  • Feature selection keeps a subset of the original inputs and drops the rest.
  • Irrelevant features raise training accuracy but lower test accuracy.
  • Filter methods score features one at a time and miss interactions.
  • Wrapper methods such as forward selection search subsets by retraining the model.
  • Embedded methods such as the lasso and tree importance select features during training.
  • Selection must run inside cross-validation, or the performance estimate is biased upward.

Common questions

What is the difference between feature selection and dimensionality reduction?

Feature selection keeps some of the original columns unchanged. Dimensionality reduction, such as PCA or an autoencoder, builds new features as combinations of all the original ones. Selection keeps the model interpretable and can remove columns from data collection entirely, while reduction can capture signal spread across many weak features.

How many features should I keep?

Treat the number as a hyperparameter and choose it with cross-validation, scoring each candidate count on held-out folds. A common practical rule is to pick the smallest set whose score is within one standard error of the best. Domain constraints, such as the cost of collecting each feature, can justify a smaller set.

How should correlated features be handled?

Two highly correlated features carry mostly the same information, so selection methods may keep either one, and the choice can flip between runs. The lasso tends to pick one arbitrarily, while the elastic net, which mixes L1 and L2 penalties, keeps correlated groups together. Tree importance splits credit between them, so each can look less important than it is.

Is feature selection needed for deep learning?

Less often on raw data such as images, audio, and text, because networks learn their own representations and are trained on large datasets. It still helps on tabular data with many noisy columns, where removing junk inputs can improve accuracy and cut serving costs. Regularization and dropout handle part of the same job inside the network.

Related explainers

References

  1. Guyon & Elisseeff (2003), "An Introduction to Variable and Feature Selection", JMLR
  2. Tibshirani (1996), "Regression Shrinkage and Selection via the Lasso", JRSS-B
  3. Breiman (2001), "Random Forests", Machine Learning
  4. Ambroise & McLachlan (2002), "Selection bias in gene extraction on the basis of microarray gene-expression data", PNAS

Last updated