Hyperparameters

Parameters (weights, biases) are learned during training — the model figures them out automatically. Hyperparameters (learning rate, layer size, batch size) are set by you before training begins. With 4 hidden neurons, this network has 26 learnable parameters (weights + biases).

A good learning rate lets loss decrease steadily and converge to a low value. The sweet spot is usually between 0.01 and 1.0 -- but the optimal value depends on the problem.

Batch size 32 is a mini-batch. This balances gradient noise with stability -- the most common choice in practice. Moderate noise helps generalization.

With 2 hidden layers and 6 neurons each, the network has 80 parameters. Each layer builds higher-level abstractions on top of the previous one.

With lambda near 0, no regularization. The model overfits -- training loss is low but validation loss climbs as the model memorizes noise. The gap between curves grows over time.

Random search samples hyperparameters uniformly at random. It often finds better configurations than grid search because it explores more unique values along each dimension. With 20 trials, the best loss found is 0.010. Research shows random search is more efficient when some hyperparameters matter more than others.