Weak Law of Large Numbers

Statement

Let $X_1, X_2, \ldots$ be i.i.d. random variables with finite mean $\mu$ and finite variance $\sigma^2$. Define the sample mean:

$\bar{X}_n = \frac{1}{n} \sum_{i=1}^n X_i$

Theorem (Weak Law of Large Numbers): For any $\epsilon > 0$:

$\lim_{n \to \infty} P(|\bar{X}_n - \mu| > \epsilon) = 0$

The sample mean converges in probability to the true mean.

Interpretation

As we collect more data, the sample mean gets arbitrarily close to the true mean with high probability. This justifies using $\bar{X}_n$ to estimate $\mu$.

“Power of Averaging”: Repeated samples allow us to estimate $\mu$ with increasing precision.

Proof (Using Chebyshev)

$E[\bar{X}_n] = \mu, \quad \text{Var}(\bar{X}_n) = \frac{\sigma^2}{n}$

By Chebyshev’s inequality:

$P(|\bar{X}_n - \mu| \geq \epsilon) \leq \frac{\text{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}$

As $n \to \infty$: $\frac{\sigma^2}{n\epsilon^2} \to 0$.

Therefore:

$\lim_{n \to \infty} P(|\bar{X}_n - \mu| \geq \epsilon) = 0$

Convergence in Probability

The WLLN says $\bar{X}_n \xrightarrow{P} \mu$ (converges in probability).

This means: for any $\epsilon > 0, \delta > 0$, there exists $N$ such that for all $n \geq N$:

$P(|\bar{X}_n - \mu| < \epsilon) > 1 - \delta$

Example: Coin Flips

Flip a fair coin $n$ times. Let $X_i = 1$ if heads, 0 otherwise. $\mu = 0.5$.

The sample proportion $\bar{X}_n$ estimates the true probability.

For $n = 100$:

$P(|\bar{X}_{100} - 0.5| \geq 0.1) \leq \frac{0.25}{100 \times 0.01} = \frac{0.25}{1} = 0.25$

At most 25% chance the sample proportion deviates by more than 0.1 from 0.5.

For $n = 10000$:

$P(|\bar{X}_{10000} - 0.5| \geq 0.1) \leq 0.0025$

Only 0.25% chance of such deviation.

What WLLN Does NOT Say

1. Not “almost all” sequences: WLLN doesn’t say $\bar{X}_n$ will eventually stay close to $\mu$ for almost every sequence (that’s the Strong Law)

2. Doesn’t give rate: WLLN says probability goes to 0, but not how fast (CLT gives the rate)

3. Requires finite variance: The proof using Chebyshev needs $\sigma^2 < \infty$ (though WLLN holds with only finite mean)

Visualisation

As $n$ increases, the distribution of $\bar{X}_n$ becomes more concentrated around $\mu$.

Weaker Conditions

WLLN holds under weaker conditions:

• Finite mean suffices (proof more subtle)
• Variance need not be finite

Historical Quote

“For even the most stupid of men, by some instinct of nature, by himself and without any instruction (which is a remarkable thing), is convinced that the more observations have been made, the less danger there is of wandering from one’s goal.” — Jacob Bernoulli (1655-1705)

Summary

Property	Statement
Assumptions	$X_i$ i.i.d. with finite $\mu$, $\sigma^2$
Conclusion	$\bar{X}_n \xrightarrow{P} \mu$
Rate	$P(
Application	Justifies sample mean as estimator