Stationary Points

Definition

A stationary point of $f(x, y)$ occurs where:

$\nabla f = \mathbf{0}$

That is: $\frac{\partial f}{\partial x} = 0$ and $\frac{\partial f}{\partial y} = 0$.

At a stationary point, the function is “flat” in all directions.

---

Types of Stationary Points

1. Local minimum: $f$ increases in all directions
2. Local maximum: $f$ decreases in all directions
3. Saddle point: $f$ increases in some directions, decreases in others

---

The Hessian Matrix

At a stationary point, the second-order behaviour is determined by the Hessian matrix:

$H = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{pmatrix}$

Since $f_{xy} = f_{yx}$ (for smooth functions), $H$ is symmetric.

---

Classification Criterion

Let $D = \det(H) = f_{xx}f_{yy} - (f_{xy})^2$

Condition	Classification
$D > 0$ and $f_{xx} > 0$	Local minimum
$D > 0$ and $f_{xx} < 0$	Local maximum
$D < 0$	Saddle point
$D = 0$	Inconclusive

Why $D > 0$ implies extremum: Both eigenvalues of $H$ have same sign.

Why $D < 0$ implies saddle: Eigenvalues have opposite signs.

Why $D = 0$ is inconclusive: One eigenvalue is zero; higher-order terms needed.

---

Taylor Series Interpretation

Near stationary point $(a, b)$:

$f(a+h, b+k) \approx f(a,b) + \frac{1}{2}(h^2 f_{xx} + 2hk f_{xy} + k^2 f_{yy})$

In matrix form: $f \approx f(a,b) + \frac{1}{2}\mathbf{h}^T H \mathbf{h}$

The quadratic form $\mathbf{h}^T H \mathbf{h}$ determines whether we have:

• Minimum: quadratic positive in all directions ($H$ positive definite)
• Maximum: quadratic negative in all directions ($H$ negative definite)
• Saddle: quadratic can be positive or negative depending on direction

---

Example

Find and classify stationary points of $f(x, y) = x^3 - 4x^2 + 2xy - y^2$.

Find stationary points:

• $f_x = 3x^2 - 8x + 2y = 0$
• $f_y = 2x - 2y = 0$

From $f_y$: $y = x$.

Substituting into $f_x$: $3x^2 - 8x + 2x = 0$ $3x^2 - 6x = 0$ $x(3x - 6) = 0$

So $x = 0$ or $x = 2$.

Stationary points: $(0, 0)$ and $(2, 2)$.

Compute Hessian:

• $f_{xx} = 6x - 8$
• $f_{xy} = 2$
• $f_{yy} = -2$

At $(0, 0)$: $H = \begin{pmatrix} -8 & 2 \\ 2 & -2 \end{pmatrix}$

$D = (-8)(-2) - 4 = 12 > 0$, and $f_{xx} = -8 < 0$.

Local maximum.

At $(2, 2)$: $H = \begin{pmatrix} 4 & 2 \\ 2 & -2 \end{pmatrix}$

$D = (4)(-2) - 4 = -12 < 0$.

Saddle point.

---

Global vs Local

• Local minimum: $f(x,y) \geq f(a,b)$ for all nearby $(x,y)$
• Global minimum: $f(x,y) \geq f(a,b)$ for all $(x,y)$ in the domain

Multiple stationary points may exist; the global minimum/maximum is found by comparing function values.

---

Connection to Eigenvalues

The Hessian is real symmetric, so has real eigenvalues $\lambda_1, \lambda_2$.

• Both positive → minimum ($D > 0, f_{xx} > 0$)
• Both negative → maximum ($D > 0, f_{xx} < 0$)
• One positive, one negative → saddle ($D < 0$)

$D = \lambda_1\lambda_2$ and $f_{xx}$ determines the sign of $\lambda_1$ (for convention).

---

Why Stationary Points Matter

• Optimisation problems (minimise cost, maximise efficiency)
• Machine learning (gradient descent finds minima of loss functions)
• Physics (equilibrium states are stationary points of potential energy)
• Economics (profit maximisation, cost minimisation)