Mean Squared Error

Definition

The mean squared error (MSE) of an estimator $\hat{\theta}$ for parameter $\theta$ is:

$\text{MSE}(\hat{\theta}) = E[(\hat{\theta} - \theta)^2]$

Bias-Variance Decomposition

$\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + \text{Bias}(\hat{\theta})^2$

Proof:

$E[(\hat{\theta} - \theta)^2] = E[((\hat{\theta} - E[\hat{\theta}]) + (E[\hat{\theta}] - \theta))^2]$

$= E[(\hat{\theta} - E[\hat{\theta}])^2] + (E[\hat{\theta}] - \theta)^2 + 2E[(\hat{\theta} - E[\hat{\theta}])(E[\hat{\theta}] - \theta)]$

The cross-term vanishes: the second bracket is constant.

$= \text{Var}(\hat{\theta}) + \text{Bias}(\hat{\theta})^2$

Interpretation

MSE measures total error:

• Variance: Random error from sampling
• Bias$^2$: Systematic error from estimator design

A good estimator balances both.

For Unbiased Estimators

If $\hat{\theta}$ is unbiased ($\text{Bias} = 0$):

$\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta})$

MSE equals variance.

Example: Sample Mean

$\bar{X}$ is unbiased for $\mu$:

$\text{MSE}(\bar{X}) = \text{Var}(\bar{X}) = \frac{\sigma^2}{n}$

Example: Sample Variance Variants

$S_n^2 = \frac{1}{n}\sum(X_i - \bar{X})^2$: biased, lower variance.

$S^2 = \frac{1}{n-1}\sum(X_i - \bar{X})^2$: unbiased, higher variance.

Which has lower MSE? For normal distribution, $S_n^2$ often wins.

Root Mean Squared Error

$\text{RMSE} = \sqrt{\text{MSE}}$

RMSE is in the same units as $\theta$, making it more interpretable.

Consistent Estimators

If $\hat{\theta}_n$ is consistent, then $\text{MSE}(\hat{\theta}_n) \to 0$ as $n \to \infty$.

Consistency requires vanishing MSE.

Optimality

Among unbiased estimators, the one with minimum variance (and hence minimum MSE) is called efficient or minimum variance unbiased estimator (MVUE).

Summary

Quantity	Formula
MSE	$E[(\hat{\theta} - \theta)^2]$
Decomposition	$\text{Var}(\hat{\theta}) + \text{Bias}^2$
RMSE	$\sqrt{\text{MSE}}$