1Classification vs Regression 2Loss Functions 3Gradient Descent 4Feature Scaling 5Hyperparameters 6Epochs & Batching

Lensa ML

Hyperparameters

Step 1 of 6: Parameters vs Hyperparameters

What the model learns vs what you choose

Hidden neurons4

Parameters

Hyperparams

Hidden size

Parameters (weights, biases) are learned during training. Hyperparameters (learning rate, layer size, batch size) are set by you before training begins. With 4 hidden neurons, this network has 26 learnable parameters (weights + biases).

Drag Hidden neurons from 2 to 10 and watch the parameter count. With 3 inputs, h hidden neurons, and 2 outputs, the network has (3h + h) + (2h + 2) = 6h + 2 parameters: 26 at h = 4 and 62 at h = 10. A simple test separates the two kinds: if gradient descent updates it, it is a parameter. The optimizer, the number of epochs, the dropout rate, and the activation function are all hyperparameters too. They are chosen on a validation set, never on the test set.

Step 2 of 6: Learning Rate

The most important hyperparameter in deep learning

Adjust learning rate (log scale)0.071

Learning rate

0.071

Final loss

1.083

Status

Too slow

The learning rate is on the low side: after 40 epochs the loss (1.08) is still falling. It would get there with more epochs or a larger rate.

Drag the learning rate slider, which runs on a log scale from 0.001 to 5. A log scale fits because useful values span several orders of magnitude, and going from 0.001 to 0.002 matters as much as going from 0.1 to 0.2. These loss curves are simulated: the status reads Too slow below about 0.2 and Diverged above 2. Real training rarely keeps one rate throughout; a warmup followed by a decay schedule is common. When the batch size grows by a factor k, the linear scaling rule multiplies the learning rate by k as well.

Step 3 of 6: Batch Size

How many samples per gradient update

Batch size32

Batch size

Grad noise

Medium

Steps/epoch

Batch size 32 is a mini-batch. It balances gradient noise against stability, which makes it the most common choice in practice.

Move the Batch size slider through 1, 8, 32, 128, and 512. With 1,024 training samples, that gives 1,024, 128, 32, 8, or 2 updates per epoch. The gradient noise falls as 1/√B, so going from 32 to 128 halves it. In practice, GPU memory caps the batch size, and powers of 2 are a convention rather than a rule. Very large batches can generalize worse, an effect linked to sharp minima. Larger batches usually need a larger learning rate to make the same progress per epoch.

Step 4 of 6: Network Depth & Width

How many layers, and how wide each one is

Hidden layers2

Neurons per hidden layer6

Total params

Layers

Width

With 2 hidden layers and 6 neurons each, the network has 80 parameters. Each layer builds on the features computed by the one before it.

Set Hidden layers from 1 to 5 and Neurons per hidden layer from 2 to 16. The default 2 layers of 6 neurons has 80 parameters, and 5 layers of 16 has 1,186. Each hidden-to-hidden layer adds w² + w parameters, so doubling the width roughly quadruples them, while adding a layer adds them linearly. Real architectures are far larger: ResNet-50 has about 25 million parameters across 50 layers. Depth and width are usually copied from a proven architecture and then scaled, not searched from scratch.

Step 5 of 6: Regularization Strength

Balancing underfitting and overfitting with lambda

L2 regularization strength (lambda)0.00

Lambda

0.00

Train loss

0.090

Val loss

1.140

Gap

1.050

With lambda = 0.00, there is too little regularization. The model overfits: training loss stays low while validation loss climbs as the model memorizes noise.

Drag the L2 regularization strength (lambda) from 0 to 1. The label reads Overfit below 0.28, Good up to 0.68, and Underfit above. L2 regularization adds λΣw² to the loss, so each gradient step also shrinks every weight by a factor of (1 − 2αλ). That is why it is called weight decay. These curves are simulated to show the three regimes. Real values of λ are usually much smaller, around 0.0001 to 0.01, and are searched on a log scale. AdamW applies weight decay separately from the Adam update.

Step 6 of 6: The Search

Grid search vs random search for hyperparameter tuning

Number of trials20

Best loss

0.077

Trials

Method

Random

Random search samples hyperparameters uniformly at random, so every trial tries a new value along each axis. Here the loss depends mostly on the learning rate, so those extra learning rates pay off. With 20 trials, the best loss found is 0.077.

Change Number of trials from 5 to 50 and switch between Grid Search and Random Search. The axes are learning rate and batch size, scaled to 0–1. With 20 trials, the grid fills four rows of a 5×5 layout, so it tests only 5 learning rates, while random search tests 20. If only the learning rate matters, random search gets four times the coverage. Here the loss depends almost entirely on the learning rate, through a narrow valley near 0.275, and barely on batch size. At 20 trials the grid's best loss is 0.212 and random search reaches 0.077, and random search finds the lower loss at every budget from 5 to 50 trials. Bayesian optimization and Hyperband spend the budget more efficiently still.

What are hyperparameters in machine learning?

Hyperparameters are the settings chosen before training that control how a model learns, such as the learning rate, batch size, number of layers, and regularization strength. They differ from parameters, the weights and biases that training learns from data. Training cannot adjust a hyperparameter directly; the only way to judge one is to train with it and measure the result.

The right hyperparameters can decide whether a network trains at all. A learning rate ten times too high makes the loss diverge, and too little regularization lets a large model memorize its training set. Practitioners tune them by training several models and comparing their scores on a validation set, using grid search, random search, or Bayesian optimization.

Key ideas

Parameters are learned by training; hyperparameters are chosen before it.
The learning rate is usually the most important hyperparameter to tune.
Batch size trades gradient noise against the cost of each update.
Depth and width set how many parameters a network has.
Regularization strength balances overfitting against underfitting.
Random search usually beats grid search when only a few hyperparameters matter.

Common questions

Which hyperparameters should be tuned first?

Start with the learning rate, since it has the largest effect and interacts with almost everything else. Then tune regularization, such as weight decay and dropout, and the batch size. Architecture choices are usually taken from a known model and adjusted last.

Why must hyperparameters be tuned on a validation set?

Each comparison between settings uses the data it is scored on to make a choice. If that data is the test set, the final test score becomes optimistically biased. A separate validation set, or cross-validation, keeps the test set as an unbiased estimate of performance on new data.

What is Bayesian optimization for hyperparameters?

It fits a probabilistic model, often a Gaussian process, to the scores of past trials and uses it to choose the next setting worth trying. It balances trying settings near the best result so far with exploring regions the model is unsure about. It typically needs fewer trials than random search when each training run is expensive.

Are hyperparameters the same for every dataset?

No. Good defaults transfer reasonably well within a model family, such as a learning rate near 0.001 for Adam, but the best values depend on the dataset size, the model, and the batch size. A larger dataset usually needs less regularization, for example.

Related explainers

Gradient Descent — Optimizing by following the slope downhill
Learning Rate Scheduling — Step decay, cosine annealing, warmup, and the LR range test
Regularization — Dropout, L1, and L2 — keeping models honest
Epochs & Batching — How data is fed to the model in epochs, batches, and shuffles
Cross-Validation — K-fold, stratified, and leave-one-out evaluation
Train, Validation & Test Sets — Splitting data to measure generalization and avoid overfitting

References

Last updated October 25, 2026