Lagrange Multipliers

Problem Statement

Find the stationary points of $f(x, y)$ subject to constraint $g(x, y) = 0$.

The constraint restricts us to a curve (or surface), so we seek constrained extrema rather than global ones.

---

Geometric Intuition

At a constrained stationary point:

• $\nabla f$ is perpendicular to the level curve of $f$ passing through that point
• $\nabla g$ is perpendicular to the constraint curve $g = 0$
• For an extremum, these level curves must be tangent

Therefore: $\nabla f$ and $\nabla g$ are parallel at the extremum.

$\nabla f = \lambda \nabla g$

where $\lambda$ is called the Lagrange multiplier.

---

Method

To find stationary points of $f(x, y)$ subject to $g(x, y) = 0$:

1. Form the Lagrangian: $\mathcal{L}(x, y, \lambda) = f(x, y) - \lambda g(x, y)$

2. Solve the system:

   • $\frac{\partial \mathcal{L}}{\partial x} = 0$
   • $\frac{\partial \mathcal{L}}{\partial y} = 0$
   • $\frac{\partial \mathcal{L}}{\partial \lambda} = 0$ (which is just $g(x, y) = 0$)

3. Evaluate $f$ at each solution to classify.

---

Example

Maximise $f(x, y) = xy$ subject to $x^2 + y^2 = 1$.

Setup: $\mathcal{L} = xy - \lambda(x^2 + y^2 - 1)$

Equations:

• $\frac{\partial \mathcal{L}}{\partial x} = y - 2\lambda x = 0$ …(i)
• $\frac{\partial \mathcal{L}}{\partial y} = x - 2\lambda y = 0$ …(ii)
• $\frac{\partial \mathcal{L}}{\partial \lambda} = -(x^2 + y^2 - 1) = 0$ …(iii)

Solve: From (i): $y = 2\lambda x$

Substitute into (ii): $x - 2\lambda(2\lambda x) = x(1 - 4\lambda^2) = 0$

So $x = 0$ or $\lambda = \pm\frac{1}{2}$.

If $x = 0$: Then $y = \pm 1$ from (iii). $f = 0$.

If $\lambda = \frac{1}{2}$: From (i), $y = x$. From (iii), $x^2 + x^2 = 1$, so $x = \pm\frac{1}{\sqrt{2}}$.

Points: $(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$ and $(-\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})$ with $f = \frac{1}{2}$.

If $\lambda = -\frac{1}{2}$: $y = -x$. Points: $(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})$ and $(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$ with $f = -\frac{1}{2}$.

Maximum occurs when $x = y = \pm\frac{1}{\sqrt{2}}$, giving $f = \frac{1}{2}$.

---

Multiple Constraints

For constraints $g_1(x, y) = 0$ and $g_2(x, y) = 0$:

$\nabla f = \lambda_1 \nabla g_1 + \lambda_2 \nabla g_2$

The Lagrangian becomes:

$\mathcal{L} = f - \lambda_1 g_1 - \lambda_2 g_2$

---

Interpretation of $\lambda$

Adding a small change $\varepsilon$ to the constraint:

$g(x, y) = \varepsilon$

The optimal value $f^*(\varepsilon)$ satisfies:

$\frac{df^*}{d\varepsilon} = \lambda$

$\lambda$ measures the sensitivity: how much the optimal value changes per unit change in the constraint.

Example: In economics, if $f$ is profit and $g$ is a budget constraint, $\lambda$ is the marginal value of relaxing the budget.

---

When Lagrange Multipliers are Needed

• Constraints cannot be easily substituted into the objective
• Multiple constraints interact
• Need to understand the “cost” of constraints (via $\lambda$)
• Generalising to higher dimensions (constrained on surfaces)

When a single constraint can be substituted, that’s often simpler. Lagrange multipliers are a systematic method for complex situations.

---

Why This Method Matters

• Optimisation: Constraints arise naturally in real problems
• Economics: Utility maximisation under budget constraints
• Physics: Equilibrium under constraints (frictionless motion)
• Machine learning: Regularisation as constrained optimisation