Central Limit Theorem

Motivation

The Laws of Large Numbers say $\bar{X}_n$ converges to $\mu$. But how fast, and what is the distribution of $\bar{X}_n$ for large $n$?

The CLT answers: it’s approximately normal.

Statement

Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and variance $\sigma^2$ ($0 < \sigma^2 < \infty$). Define:

$\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$

Central Limit Theorem:

$\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0, 1)$

Equivalently:

$\sqrt{n}(\bar{X}_n - \mu) \xrightarrow{d} N(0, \sigma^2)$

Or:

$\sum_{i=1}^n X_i \approx N(n\mu, n\sigma^2)$

for large $n$.

Interpretation

• The sample mean $\bar{X}_n$ is approximately normal
• Mean: $\mu$
• Variance: $\sigma^2/n$ (decreases with $n$)
• Standard error: $\sigma/\sqrt{n}$

The ”$\sqrt{n}$ rate”: standard deviation shrinks as $1/\sqrt{n}$.

Normal Approximation

For large $n$:

$P\left(a \leq \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \leq b\right) \approx \Phi(b) - \Phi(a)$

This works regardless of the original distribution of $X_i$.

Conditions

1. Independence: $X_i$ are independent
2. Identical distribution: $X_i$ have the same distribution
3. Finite variance: $\sigma^2 < \infty$

The CLT works for most common distributions.

Example: Coin Flips

Flip fair coin $n = 100$ times. $X_i \in \{0, 1\}$ with $\mu = 0.5$, $\sigma^2 = 0.25$.

Total heads: $S_{100} = \sum_{i=1}^{100} X_i$.

By CLT: $S_{100} \approx N(50, 25)$, so:

$P(S_{100} \geq 60) = P\left(\frac{S_{100} - 50}{5} \geq \frac{60 - 50}{5}\right) \approx P(Z \geq 2) = 0.0228$

Exact (binomial): $P(S_{100} \geq 60) = \sum_{k=60}^{100} \binom{100}{k} 0.5^{100} \approx 0.028$.

CLT approximation is close.

Continuity Correction

When approximating discrete distributions, add 0.5:

$P(S_n \geq k) \approx P\left(Z \geq \frac{k - 0.5 - n\mu}{\sigma\sqrt{n}}\right)$

Using continuity correction in coin example:

$P(S_{100} \geq 60) \approx P\left(Z \geq \frac{59.5 - 50}{5}\right) = P(Z \geq 1.9) = 0.0287$

Better approximation: 0.0287 vs exact 0.028.

Visualisation

As $n$ increases, the distribution of $\bar{X}_n$ becomes more bell-shaped and narrower.

Why Does It Work?

Each $X_i$ contributes a “small random perturbation”. The sum of many small independent random effects tends toward normality.

This explains why many natural phenomena are normally distributed: they result from many small factors additive up.

Application: Confidence Intervals

Approximate 95% confidence interval for $\mu$:

$\bar{X}_n \pm 1.96 \frac{\sigma}{\sqrt{n}}$

where $\sigma$ is known, or replaced by sample standard deviation $s$ for large $n$.

Rate of Convergence

CLT gives the rate: $O(1/\sqrt{n})$. The error in the CLT approximation is typically $O(1/\sqrt{n})$, quantified by the Berry-Esseen theorem.

Example: Load Times

A website has average page load time $\mu = 2$ seconds, SD $\sigma = 0.5$ seconds. For $n = 25$ users:

$\bar{X}_{25} \approx N(2, 0.5^2/25) = N(2, 0.01)$

$P(\bar{X}_{25} > 2.2) = P\left(Z > \frac{2.2 - 2}{0.1}\right) = P(Z > 2) \approx 0.0228$

Summary

Aspect	Statement
Conclusion	$\sqrt{n}(\bar{X}_n - \mu) \xrightarrow{d} N(0, \sigma^2)$
Rate	$O(1/\sqrt{n})$
Approximation	$\bar{X}_n \approx N(\mu, \sigma^2/n)$
Conditions	i.i.d., finite $\sigma^2$