Bias of Estimators

Definition

The bias of an estimator $\hat{\theta}$ for parameter $\theta$ is:

$\text{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta$

Interpretation

• $\text{Bias} = 0$: unbiased
• $\text{Bias} > 0$: overestimates on average
• $\text{Bias} < 0$: underestimates on average

Unbiased Estimators

$\hat{\theta}$ is unbiased if $E[\hat{\theta}] = \theta$ for all $\theta$.

Example: Sample Mean

$E[\bar{X}] = E\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n E[X_i] = \mu$

Unbiased for population mean.

Example: Sample Variance

$S_n^2 = \frac{1}{n}\sum_{i=1}^n(X_i - \bar{X})^2$

$E[S_n^2] = \frac{n-1}{n}\sigma^2$ (biased, underestimates).

$S^2 = \frac{1}{n-1}\sum_{i=1}^n(X_i - \bar{X})^2$

$E[S^2] = \sigma^2$ (unbiased).

Why Unbiasedness Matters

Unbiased estimators:

• Average to the true value over repeated sampling
• No systematic error
• “Fair” on average

But being unbiased doesn’t guarantee good performance:

• A fair coin estimator averaged with a biased one might be worse
• An unbiased estimator with huge variance is useless

Bias-Variance Trade-off

Sometimes a slightly biased estimator has much lower variance.

$\text{MSE} = \text{Bias}^2 + \text{Variance}$

Minimising MSE may tolerate some bias for greatly reduced variance.

Example: Ridge Regression

Shrinking coefficients introduces bias but reduces variance.

Overall prediction error decreases.

Asymptotic Unbiasedness

$\hat{\theta}_n$ is asymptotically unbiased if:

$\lim_{n \to \infty} E[\hat{\theta}_n] = \theta$

$S_n^2$ is asymptotically unbiased: $E[S_n^2] = \frac{n-1}{n}\sigma^2 \to \sigma^2$.

Summary

Estimator	Bias
$\bar{X}$	0 (unbiased)
$S_n^2$	$-\sigma^2/n$ (biased)
$S^2$	0 (unbiased)