1Derivatives 2The Chain Rule 3Partial Derivatives & Gradients 4Logarithms & Exponentials 5Matrix Calculus & Jacobians 6Convexity, Minima & Saddle Points 7Constrained Optimization

Lensa ML

How Constrained Optimization Works

Step 1 of 6: Unconstrained vs Constrained

What happens when you can't reach the true minimum

Constraint radius r1.50

Unconstrained min

0.0

Constrained min

1.76

Gap

1.76

With a small radius, the circle is far from the unconstrained minimum, so the best feasible point is much worse.

Drag the Constraint radius r from 0.5 to 3 and watch the gap card. The objective is f = (x − 2)² + (y − 2)², whose free minimum is 0 at (2, 2), and the point must stay on the circle x² + y² = r². The best point on the circle lies on the line toward (2, 2), so the constrained minimum is (r − 2√2)². At r = 1.5 that is about 1.76. The gap is the price of the constraint, and it is never negative.

Step 2 of 6: The Constraint Curve

Exploring the feasible region for better solutions

Constraint radius r1.50

Position t along constraint0.79

1.06

1.07

f(x,y)

1.76

Optimal f

1.76

This is at or very near the constrained optimum, the best value of f anywhere on the circle.

Set the radius, then move Position t along constraint around the full circle from 0 to 2π. Writing the point as (r·cos t, r·sin t) turns a two-variable constrained problem into a one-variable unconstrained one, which is the substitution method. It works when the constraint is easy to parameterize, like a circle. For most real constraints no such formula exists, which is why the gradient condition in the next steps is more useful.

Step 3 of 6: Gradient Alignment

At the optimum, objective and constraint gradients align

Position t along constraint0.79

|∇f|

2.66

|∇g|

3.00

Angle

179.4°

f(x,y)

1.76

The gradients are nearly parallel, which is the optimality condition. At the constrained minimum, ∇f and ∇g point in the same or opposite directions.

Move Position t along constraint and watch the ANGLE card. Here r is fixed at 1.5. Moving along the circle means moving in the tangent direction, and f changes at a rate equal to the component of ∇f along that tangent. When ∇f has no tangent component, it is perpendicular to the circle, so it must lie along ∇g = (2x, 2y). At this optimum the two arrows point in opposite directions, because the free minimum (2, 2) lies outside the circle.

Step 4 of 6: The Lagrange Multiplier λ

The scalar that makes ∇f = λ∇g

Position t along constraint0.79

Lagrange multiplier λ1.00

1.0

|∇f − λ∇g|

5.66

Optimal λ

-0.89

Large mismatch: this λ does not satisfy ∇f = λ∇g. Move λ toward the value in the green card.

Set t near π/4 ≈ 0.79 and tune the Lagrange multiplier λ until the mismatch card turns green. Solving 2(x − 2) = λ·2x on the optimal point gives λ = 1 − 2√2/r, about −0.89 for r = 1.5. The multiplier has a meaning: it is the rate at which the best objective value changes as the constraint level r² changes. Loosening the constraint by 0.1 here lowers the optimal f by about 0.09.

Step 5 of 6: Finding the Optimum

Iteratively converging to the constrained solution

Iteration step0

Step

-1.50

0.00

f(x,y)

16.25

The start is on the far side of the circle. Each iteration moves the point along the circle toward the optimum.

Drag the Iteration step slider from 0 to 20 and follow the trail from t = π around to the optimum at t = π/4. Projected gradient descent is one way to solve such problems: take an ordinary gradient step, then project the point back onto the feasible set. For a circle, projection just rescales the point to length r. Here each step uses a learning rate of 0.08, so the point covers most of the arc in the first 7 iterations and then creeps the last few degrees. Other methods include penalty methods, which add a growing cost for violations, and interior-point methods for convex problems with many constraints.

Step 6 of 6: Regularization as a Constraint

Constraining weights to prevent overfitting

Constraint budget c = ||w||² ≤ c2.00

||w||²

2.00

Loss

3.06

Budget c

2.0

Implied λ

1.24

A moderate budget pulls the weights toward the origin but still lets them fit the data reasonably well. Implied λ is the L2 penalty strength that gives the same w*: minimizing loss + λ||w||² lands on this point. It is 0 whenever the constraint is inactive.

Raise the Constraint budget c from 0.5 to 5. The loss is centered at (3, 1), which lies outside every circle in this range, so the constraint is always active and the solution is the point of the circle closest to (3, 1). Minimizing loss subject to |w|² ≤ c gives the same solution as minimizing loss + λ|w|² for some λ ≥ 0. A smaller budget corresponds to a larger λ. The KKT conditions add that λ = 0 whenever the constraint is not active.

What is constrained optimization?

Constrained optimization finds the minimum or maximum of a function when the inputs must satisfy extra conditions, such as staying on a curve or within a budget. The classic tool is the method of Lagrange multipliers: at a constrained optimum, the gradient of the objective is a multiple of the gradient of the constraint, ∇f = λ∇g. For inequality constraints this generalizes to the Karush–Kuhn–Tucker (KKT) conditions.

In machine learning, constraints show up in support vector machines (maximize the margin subject to classifying every point correctly), in weight-norm limits and L2 regularization, in fairness and safety constraints, and in the KL-divergence trust region used by TRPO and in RLHF. Many penalty terms in loss functions are constraints written in Lagrangian form.

Key ideas

A constraint limits where the solution can be, and it can only make the best objective value worse or equal.
At a constrained optimum, ∇f is parallel to the constraint gradient ∇g.
The Lagrange condition ∇f = λ∇g turns a constrained problem into a system of equations.
The multiplier λ is the sensitivity of the optimal value to the constraint level.
Inequality constraints follow the KKT conditions, where λ is zero unless the constraint is active.
L2 regularization is the penalty form of a constraint on the weight norm.

Common questions

What is the Lagrangian?

The Lagrangian combines the objective and constraints into one function: L(x, λ) = f(x) − λ·(g(x) − c). Setting its gradient to zero with respect to both x and λ recovers ∇f = λ∇g and the constraint itself. Sign conventions vary between textbooks, so λ may appear with a plus sign.

What are the KKT conditions?

They are the optimality conditions for problems with inequality constraints. They require stationarity of the Lagrangian, feasibility, a non-negative multiplier for each inequality, and complementary slackness: each multiplier times its constraint slack equals zero. For convex problems that satisfy a mild regularity condition, the KKT conditions are both necessary and sufficient.

How do support vector machines use constrained optimization?

A hard-margin SVM minimizes |w|²/2 subject to every training point being on the correct side of the margin. Its Lagrange dual depends on the data only through dot products between points, which is what allows the kernel trick. Points with nonzero multipliers are the support vectors.

How are constraints handled in neural network training?

Usually through penalties added to the loss, such as weight decay, or through projection after each step, such as clipping weights or renormalizing them to a maximum norm. Methods like TRPO enforce a trust-region constraint on the policy change directly. Exact constrained solvers are rare at deep learning scale.

Related explainers

Convexity, Minima & Saddle Points — Why some optimization problems are harder than others
Partial Derivatives & Gradients — Extend single-variable derivatives to functions of multiple variables and gradient vectors
Regularization — Dropout, L1, and L2 — keeping models honest
Support Vector Machines — Maximum margin classifiers and the kernel trick
Gradient Descent — Optimizing by following the slope downhill
Norms & Distance Metrics — L1, L2, cosine similarity — measuring closeness in vector spaces

References

Last updated October 25, 2026