1Classification vs Regression 2Loss Functions 3Gradient Descent 4Feature Scaling 5Hyperparameters 6Epochs & Batching

Lensa ML

Epochs & Batching

Step 1 of 6: What Is an Epoch?

One complete pass through the entire dataset

Jump to sampleSample 1

Epoch

Current sample

1 / 16

Total seen

An epoch is one complete pass through every sample in the dataset. The scanner visits each sample once per epoch. After all 16 samples are seen, epoch 1 is complete and the next one begins.

Press Play or Step Once, or drag Jump to sample, to move the scanner across the 16 samples. Three terms are easy to mix up. An epoch is a full pass over the data, a batch is the group of examples used for one update, and an iteration is the update itself. ImageNet's 1.28 million training images in batches of 256 give about 5,005 iterations per epoch. Large language models often train for about one epoch over their text, so their progress is counted in tokens seen instead.

Step 2 of 6: Stochastic Gradient Descent

Updating weights one sample at a time

Learning rate0.080

Step

Loss

15.813

Grad mag

23.412

Stochastic Gradient Descent updates weights after each single sample. The gradient estimate is noisy because one sample cannot represent the full dataset, so the path jitters and zigzags: each update pulls in a slightly different direction.

Press Play or Step Once, and change the Learning rate between 0.001 and 0.200. The loss here is 0.1x² + x²y² + 0.5y², starting at 15.8, and each step adds random noise to the true gradient to mimic using a single sample. The noise does not fade at the minimum. With a fixed learning rate, SGD keeps bouncing inside a region whose size grows with the rate, which is one reason rates are decayed late in training. Robbins and Monro analyzed this kind of noisy update in 1951.

Step 3 of 6: Mini-batch Gradient Descent

The practical middle ground between SGD and full-batch

Batch size4

Learning rate0.080

Step

Loss

15.813

Grad mag

23.412

Steps / epoch

Batch size 4 averages gradients over 4 samples per update. The path is smoother than SGD while still updating 16 times per epoch. This middle ground is what most training uses.

Move the Batch size slider through 1, 4, 16, and 64 for a dataset of 64 samples. The noise added here scales as 1.5/√B times √((64 − B)/63), so it reaches zero at the full batch of 64, and the updates per epoch fall from 64 to 1. On a GPU, a batch of 32 often takes little longer than a single example, because the batch runs through each layer as one parallel matrix multiplication. When the dataset size is not a multiple of the batch size, the last batch is smaller. PyTorch's DataLoader can discard it with drop_last=True.

Step 4 of 6: Full-batch Gradient Descent

Using the whole dataset for each update

Batch size16

Samples per step (memory)

Updates per epoch

Final loss

0.020

Full-batch gradient descent uses every sample to compute each gradient: the smoothest path, but the slowest and most memory-hungry per step. Compared with the faded full-batch reference, smaller batches are noisier but update the weights more often per epoch. The tradeoff: memory vs speed.

Move the Batch size slider and compare the colored path with the faded full-batch reference. Every path here takes 80 steps, and the final loss is 0.152 at batch size 1 and about 0.02 at 4, 16, and 64. The comparison is per step, though. Per epoch, batch size 64 makes 1 update while batch size 1 makes 64. Full-batch methods suit small datasets that fit in memory and second-order optimizers such as L-BFGS. Activation memory grows with batch size, which is what usually limits it.

Step 5 of 6: Shuffling Matters

Why randomizing data order improves training

Jump to epochEpoch 1

Shuffled loss

2.200

Unshuffled loss

2.200

Epoch

Shuffling the dataset each epoch gives every mini-batch a varied mix of samples, in a new order each time. The loss curve is smoother and converges to a lower value.

Switch between Shuffled and Unshuffled, then press Play or drag Jump to epoch across the 50 epochs. These curves are simulated, and at epoch 50 the shuffled loss is 0.204 against 0.704 for the unshuffled run, caught on a spike. The classic failure is data sorted by label. A batch of only cats, then only dogs, pulls the weights back and forth between classes. Drawing a fresh permutation each epoch is the standard fix. Time series are the exception: shuffling across time before the train-test split leaks future data into training.

Step 6 of 6: Putting It Together

Epochs times batch size equals training

Epochs10

Batch size16

Total updates

Samples seen

Final loss

2.500

Training = 10 epochs x 4 steps/epoch = 40 total weight updates. Each epoch processes all 64 samples in batches of 16. Over 10 epochs, the model sees 640 total samples. Whether this is enough depends on the task; the validation loss is the usual guide.

Set Epochs from 1 to 50 and Batch size from 1 to 64, then press Play. The default, 10 epochs with batches of 16, gives 10 × 4 = 40 updates and 640 samples seen. In this simulation the loss falls per update, so after 10 epochs batch size 1 reaches 0.045 while batch size 64 is still at 1.53. Both have seen the same 640 samples. Fine-tuning a pretrained model often needs only a few epochs; the BERT paper suggested 2 to 4.

What is an epoch and batch size in machine learning?

An epoch is one full pass of a model through its entire training dataset. The batch size is the number of examples the model processes before each weight update. One epoch therefore contains the dataset size divided by the batch size updates, rounded up, and each update is called an iteration or step.

Epochs and batch size together set how many updates training makes and how noisy each one is. Deep learning frameworks such as PyTorch and TensorFlow organize training as a loop over epochs, with an inner loop over shuffled mini-batches. Choosing these two values affects training time, GPU memory use, and how well the final model generalizes.

Key ideas

An epoch is one complete pass through the training data.
The batch size is how many examples are averaged for each weight update.
Updates per epoch equal the dataset size divided by the batch size, rounded up.
Smaller batches give more, noisier updates; larger batches give fewer, smoother ones.
Shuffling every epoch keeps mini-batches representative of the whole dataset.
The number of epochs is usually set by watching the validation loss.

Common questions

How many epochs are enough?

There is no fixed number. Track the validation loss after each epoch and stop when it has not improved for several epochs, keeping the best checkpoint. This is called early stopping, and the patience setting controls how many epochs without improvement to allow.

Does a larger batch size make training faster?

Each epoch usually finishes faster, because a GPU processes a large batch more efficiently than many small ones. The model also gets fewer updates per epoch, so it may need more epochs, a larger learning rate, or both to reach the same loss. Past a certain size, extra batch size stops reducing the number of updates needed.

What is gradient accumulation?

Gradient accumulation adds up the gradients from several small batches and applies one update after the last of them. It reproduces the effect of a larger batch when the larger batch does not fit in GPU memory. Training with 4 accumulated batches of 8 behaves much like training with batches of 32, though batch normalization statistics still use the smaller batches.

Should validation and test data be shuffled?

It is not necessary. Evaluation does not update the weights, so the order of examples does not change the metrics. Shuffling matters for training only, where the order decides which examples share a batch and in what sequence the updates happen.

Related explainers

Gradient Descent — Optimizing by following the slope downhill
Hyperparameters — The knobs you tune before training begins
Optimizers — SGD, momentum, and Adam — smarter gradient descent
Learning Rate Scheduling — Step decay, cosine annealing, warmup, and the LR range test
Overfitting & Underfitting — The bias-variance tradeoff and generalization
Train, Validation & Test Sets — Splitting data to measure generalization and avoid overfitting

References

Last updated October 25, 2026