Epochs & Batching

An epoch is one complete pass through every sample in the dataset. The scanner processes each sample once. After all 16 samples are seen, epoch 1 is complete and the next one begins.

Stochastic Gradient Descent updates weights after each single sample. The gradient estimate is noisy because one sample cannot represent the full dataset. Notice the jittery, zigzag path -- each update pulls in a slightly different direction.

Batch size 4 averages gradients over 4 samples per update. The path is smoother than SGD while still updating 16 times per epoch. This is the sweet spot used in practice.

Full-batch gradient descent uses every sample to compute each gradient -- the smoothest path but the slowest per-step (and most memory hungry). Compared to the faded full-batch reference, smaller batches are noisier but update weights more frequently per epoch. The tradeoff: memory vs speed.

Shuffling the dataset each epoch ensures that mini-batches are diverse. The model sees samples in a different order every time, preventing it from memorizing patterns in the ordering. The loss curve is smoother and converges to a lower value.

Training = 10 epochs x 4 steps/epoch = 40 total weight updates. Each epoch processes all 64 samples in batches of 16. Over 10 epochs, the model sees 640 total samples. This is a reasonable number of epochs for most tasks.