How Constrained Optimization Works
Unconstrained vs Constrained
What happens when you can't reach the true minimum
With a small radius, the feasible region is far from the unconstrained minimum. The constraint forces a much worse solution.
Step 1 of 6: Unconstrained vs Constrained
What happens when you can't reach the true minimum
With a small radius, the feasible region is far from the unconstrained minimum. The constraint forces a much worse solution.
Step 2 of 6: The Constraint Curve
Exploring the feasible region for better solutions
You're very close to the constrained optimum! This is the best you can do while staying on the circle.
Step 3 of 6: Gradient Alignment
At the optimum, objective and constraint gradients align
The gradients are nearly parallel — this is the optimality condition! At the constrained minimum, ∇f and ∇g point in the same (or opposite) direction.
Step 4 of 6: The Lagrange Multiplier λ
The scalar that makes ∇f = λ∇g
Large mismatch — the current λ doesn't satisfy ∇f = λ∇g. Try moving λ toward the optimal value shown in the green card.
Step 5 of 6: Finding the Optimum
Iteratively converging to the constrained solution
Starting from the opposite side of the constraint circle. The solver will move along the constraint toward the optimum using projected gradient descent.
Step 6 of 6: Regularization as a Constraint
Constraining weights to prevent overfitting
A balanced constraint — the weights are pulled toward the origin but can still fit the data reasonably. This sweet spot trades off loss and model complexity.