Maximum Likelihood Estimation

Intuition

Choose the parameter value that makes the observed data most probable.

Likelihood Function

For data $X_1, \ldots, X_n$ and parameter $\theta$:

$L(\theta) = \prod_{i=1}^n f(X_i; \theta)$

where $f(x; \theta)$ is the PMF or PDF.

Log-Likelihood

$\ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log f(X_i; \theta)$

Easier to work with: products become sums.

MLE Definition

$\hat{\theta}_{MLE} = \arg\max_\theta L(\theta) = \arg\max_\theta \ell(\theta)$

Example: Bernoulli

$X_i \sim \text{Bern}(p)$. Data: $x_1, \ldots, x_n$.

$L(p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{\sum x_i}(1-p)^{n-\sum x_i}$

$\ell(p) = (\sum x_i)\log p + (n - \sum x_i)\log(1-p)$

Differentiate and set to 0:

$\frac{\sum x_i}{p} - \frac{n - \sum x_i}{1-p} = 0$

$\hat{p} = \frac{1}{n}\sum_{i=1}^n x_i = \bar{X}$

Example: Normal Mean (Known Variance)

$X_i \sim N(\mu, \sigma^2)$ with $\sigma$ known.

$\ell(\mu) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum(x_i - \mu)^2$

Maximise by minimising $\sum(x_i - \mu)^2$:

$\hat{\mu} = \bar{X}$

Properties of MLE

1. Consistent: $\hat{\theta}_{MLE} \xrightarrow{P} \theta$
2. Asymptotically normal: $\sqrt{n}(\hat{\theta}_{MLE} - \theta) \xrightarrow{d} N(0, I^{-1})$
3. Asymptotically efficient: Achieves minimum variance among all estimators
4. Invariant: If $\hat{\theta}$ is MLE of $\theta$, then $g(\hat{\theta})$ is MLE of $g(\theta)$

Summary

Step	Action
1	Write likelihood $L(\theta)$
2	Take log (optional but easier)
3	Differentiate, set to 0
4	Solve for $\hat{\theta}$