1Data Preprocessing 2Feature Extraction 3Dimensionality Reduction 4Embeddings 5Transfer Learning

Lensa ML

Feature Extraction

How raw data becomes the language of machine learning

Step 1 of 6: Data Starts Messy

Before ML can learn, it needs structured numbers — rows and columns of measurements

A dataset of 8 houses, one observation per row

Samples

8

Input features

5

Target

1

Each row of this table of data is one example (a house). Each column is a feature, a measurable property. The last column is the target, the price to predict.

Watch the eight houses fill the table row by row. This layout, one row per example and one column per measured property, is the input format for almost every classical ML algorithm. The inputs are usually written as a matrix X of shape n × d, with a separate target vector y. Here n = 8 and d = 5, and price is y. Feature extraction covers everything that turns raw records into the columns of X: choosing what to measure, deriving new columns, rescaling them, and compressing or dropping them. The next steps follow that order.

Step 2 of 6: Features Are the Vocabulary

Each column is a feature — one measurable property the model uses to make predictions

Min

950

Mean

1862.5

Max

3100

Range

2150.0

A feature is one column of the table. Numerical features like square footage have magnitude and spread. Categorical features like "pool" encode discrete groups. Each feature's distribution guides how to transform it.

Click each column tab, from Sq Ft to Pool, and compare the min, mean, and max readouts. Numerical features go into a model as numbers, but categorical ones need an encoding first. A binary column like Pool can stay 0 or 1. A column with many categories, such as neighborhood, is usually one-hot encoded: one 0/1 column per category, so the model does not read an order into arbitrary codes like 1, 2, 3. Text and images are harder, since their raw form is not a table at all. That is where learned feature extractors come in.

Step 3 of 6: Craft Better Signals

Transform raw columns into features that expose hidden patterns to the model

Formula: price ÷ sqft

Price per Sq Ft: Normalizes price by size: a 3000 sqft house at $600k is cheaper per foot than a 1000 sqft house at $300k. But this is a trap if you are predicting price: the column is computed from price, the target. A model fed this input would score almost perfectly in testing and then fail in use, because a new house has no price yet. That is target leakage. The feature is fine for analysis, or for predicting something else.

Click each transform, from Price per Sq Ft to Log Distance, and compare the source values with the new column. The first one hides a common trap: price per square foot is computed from the target, so using it as an input to predict price would leak the answer. It is fine for analysis or for predicting something else. Log transforms suit features that span orders of magnitude, like income or distance, and the +1 keeps the log defined at zero. Tree models can split a raw column anywhere, so binning helps linear models more than trees.

Step 4 of 6: Put Features on Equal Footing

Normalize values so no single feature dominates by sheer magnitude

Sq Ft ranges 950–3100 while Age is 2–50, so distances are dominated by Sq Ft

Feature scaling keeps large-magnitude features from dominating. Min-Max squeezes values into [0, 1]. Z-Score centers each feature at zero with unit standard deviation. Gradient descent, KNN, and SVM all work better on scaled features.

Click Raw, Min-Max, and Z-Score and watch the three number lines rescale. Min-max computes x' = (x − min) / (max − min), and z-score computes z = (x − μ) / σ. Both statistics must come from the training set only and then be reused on validation and test data; otherwise test information leaks into training. Min-max is sensitive to a single extreme value, because one outlier sets the range for every other point. Decision trees and random forests need no scaling, since a split threshold works the same after any monotonic rescaling.

Step 5 of 6: Compress Without Losing Meaning

Project high-dimensional data into fewer dimensions that capture the most variance

Original

Variance kept

95%

Reduced

PCA finds the axis of maximum variance, the direction along which the data spreads most, and projects every point onto it. Here PC1 keeps about 95% of the variance of the 2 standardized features in 1 number.

Watch the eight houses get projected onto the PC1 line and then laid out on a single axis below. The two features are standardized first, because PCA follows variance and unscaled Sq Ft would set the direction by itself. PC1 keeps so much here because Sq Ft and Age are strongly correlated in this data (r ≈ −0.90): bigger houses are newer. For two standardized features, PC1's share of the variance is (1 + |r|) / 2. PCA builds new features from combinations of old ones, while selection in the next step keeps original columns and drops the rest.

Step 6 of 6: Keep Only What Matters

Measure each feature's predictive power and discard the noise

Threshold15%

Kept

3

Dropped

2

Feature importance scores rank how much each feature contributes to predictions. Dropping the ones below the threshold leaves a simpler model that trains faster, is easier to read, and often overfits less. The scores shown are illustrative, not fitted to the eight houses above.

Drag the Threshold from 0% to 40% and watch features drop out, then switch to Correlation Matrix. Importance scores like these usually come from a tree ensemble, where each feature gets credit for how much its splits reduce error. Correlated features share that credit, so Beds can look weak only because Sq Ft already carries its signal. Permutation importance, which shuffles one column and measures how much the score falls, is a more reliable check. Selection belongs inside cross-validation, because choosing features on the full dataset leaks test information into the result.

What is feature extraction in machine learning?

Feature extraction is the process of turning raw data into numeric features, the input columns a model learns from. It covers choosing which properties to measure, deriving new ones from existing columns, rescaling them, and compressing many features into a few. The goal is a table where the patterns that predict the target are easy for a model to find.

Every machine learning pipeline does some form of it. Tabular models for pricing, credit scoring, or churn rely on hand-built features such as ratios and time since the last event. Text models start from word counts, TF-IDF, or learned embeddings, and image models from pixel statistics or the activations of a pretrained network. In deep learning the network learns its own features, but the choice of inputs and their scaling still affects how well it trains.

Key ideas

A feature is one measurable property of an example, stored as one column of the input table.
Feature engineering creates new columns, such as ratios, bins, and log transforms, that expose patterns to the model.
Scaling puts features on comparable ranges so distance- and gradient-based models are not dominated by large units.
PCA builds a few new features that keep most of the variance of many correlated ones.
Feature selection keeps the original columns that carry signal and drops redundant ones.
Every statistic used to build features must be fit on training data only, or the evaluation leaks.

Common questions

What is the difference between feature extraction and feature selection?

Feature extraction creates new features, either by transforming raw data or by combining existing columns, as PCA does. Feature selection keeps a subset of the existing columns unchanged. Selection keeps features interpretable, while extraction can compress more information into fewer numbers.

Is feature extraction still needed with deep learning?

Less of it is done by hand. A CNN learns edge and texture detectors from raw pixels, and a language model learns its own word representations. Pretrained networks are now often used as feature extractors themselves. On tabular data, hand-built features and gradient-boosted trees remain hard to beat.

What are common examples of feature extraction?

For text, bag-of-words counts, TF-IDF weights, and embeddings. For images, color histograms, edge descriptors such as HOG, or activations from a pretrained CNN. For audio, mel-frequency cepstral coefficients (MFCCs), and for time series, lags, rolling averages, and calendar features such as day of week.

How many features should a model have?

There is no fixed number, but more features need more examples to estimate their effects reliably. When features outnumber examples, models overfit easily, and regularization or selection becomes necessary. Adding a feature is worth it when validation performance improves, not when training performance does.

Related explainers

Feature Scaling — Why and how to normalize features for faster convergence
Feature Selection — Filter, wrapper, and embedded methods for choosing features
Dimensionality Reduction — PCA and t-SNE — seeing high-dimensional data
Data Preprocessing — Cleaning, scaling, and encoding raw data
Curse of Dimensionality — Why high-dimensional spaces break intuition and algorithms
Transfer Learning — Standing on the shoulders of pretrained giants

References

Last updated October 25, 2026