Spread in Distributions
The expectation E [ X ] E[X] E [ X ]
Example: All three distributions below have mean 3, but very different spread:
Value Dist A Dist B Dist C 1 0 1/3 0 2 0 0 0 3 1 1/3 0 4 0 0 0 5 0 1/3 1
A is concentrated at 3 (no spread)
B spreads evenly across 1, 3, 5 (medium spread)
C spreads to 3 and 5 (different spread)
We need a measure of spread.
Variance
Definition
The variance of X X X
Var ( X ) = E [ ( X − μ ) 2 ] \text{Var}(X) = E[(X - \mu)^2] Var ( X ) = E [( X − μ ) 2 ]
where μ = E [ X ] \mu = E[X] μ = E [ X ]
Properties:
Variance ≥ 0 \geq 0 ≥ 0
Variance = 0 = 0 = 0 X X X
Variance increases as values spread out from the mean
Var ( X ) = E [ X 2 ] − E [ X ] 2 \text{Var}(X) = E[X^2] - E[X]^2 Var ( X ) = E [ X 2 ] − E [ X ] 2
Proof:
E [ ( X − μ ) 2 ] = E [ X 2 − 2 μ X + μ 2 ] = E [ X 2 ] − 2 μ E [ X ] + μ 2 = E [ X 2 ] − 2 μ 2 + μ 2 = E [ X 2 ] − μ 2 E[(X - \mu)^2] = E[X^2 - 2\mu X + \mu^2] = E[X^2] - 2\mu E[X] + \mu^2 = E[X^2] - 2\mu^2 + \mu^2 = E[X^2] - \mu^2 E [( X − μ ) 2 ] = E [ X 2 − 2 μ X + μ 2 ] = E [ X 2 ] − 2 μ E [ X ] + μ 2 = E [ X 2 ] − 2 μ 2 + μ 2 = E [ X 2 ] − μ 2
This formula is often easier: compute E [ X ] E[X] E [ X ] E [ X 2 ] E[X^2] E [ X 2 ]
Standard Deviation
The standard deviation is the square root of variance:
σ X = Var ( X ) \sigma_X = \sqrt{\text{Var}(X)} σ X = Var ( X )
Why take the square root? Variance has units of X 2 X^2 X 2
If X X X Var ( X ) \text{Var}(X) Var ( X ) σ X \sigma_X σ X
Example: Fair Die
E [ X ] = 7 2 , E [ X 2 ] = 91 6 E[X] = \frac{7}{2}, \quad E[X^2] = \frac{91}{6} E [ X ] = 2 7 , E [ X 2 ] = 6 91
Var ( X ) = 91 6 − 49 4 = 182 − 147 12 = 35 12 ≈ 2.92 \text{Var}(X) = \frac{91}{6} - \frac{49}{4} = \frac{182 - 147}{12} = \frac{35}{12} \approx 2.92 Var ( X ) = 6 91 − 4 49 = 12 182 − 147 = 12 35 ≈ 2.92
σ X = 35 12 ≈ 1.71 \sigma_X = \sqrt{\frac{35}{12}} \approx 1.71 σ X = 12 35 ≈ 1.71
Properties of Variance
1. Constants
Var ( c ) = 0 \text{Var}(c) = 0 Var ( c ) = 0
Var ( X + c ) = Var ( X ) \text{Var}(X + c) = \text{Var}(X) Var ( X + c ) = Var ( X )
Adding a constant shifts the distribution but doesn’t change spread.
2. Scaling
Var ( a X ) = a 2 Var ( X ) \text{Var}(aX) = a^2 \text{Var}(X) Var ( a X ) = a 2 Var ( X )
Proof: Var ( a X ) = E [ ( a X ) 2 ] − E [ a X ] 2 = a 2 E [ X 2 ] − a 2 E [ X ] 2 = a 2 Var ( X ) \text{Var}(aX) = E[(aX)^2] - E[aX]^2 = a^2 E[X^2] - a^2 E[X]^2 = a^2 \text{Var}(X) Var ( a X ) = E [( a X ) 2 ] − E [ a X ] 2 = a 2 E [ X 2 ] − a 2 E [ X ] 2 = a 2 Var ( X )
3. Sum of Independent Variables
If X X X Y Y Y
Var ( X + Y ) = Var ( X ) + Var ( Y ) \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) Var ( X + Y ) = Var ( X ) + Var ( Y )
Warning: This requires independence! Without it:
Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 Cov ( X , Y ) \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y) Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 Cov ( X , Y )
Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 E [ ( X − E [ X ] ) ( Y − E [ Y ] ) ] \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2 E[(X - E[X])(Y - E[Y])] Var ( X + Y ) = Var ( X ) + Var ( Y ) + 2 E [( X − E [ X ]) ( Y − E [ Y ])]
The last term is twice the covariance.
Example: Sums of I.I.D. Variables
Let X 1 , X 2 , … , X n X_1, X_2, \ldots, X_n X 1 , X 2 , … , X n σ 2 \sigma^2 σ 2 Var ( ∑ X i ) \text{Var}(\sum X_i) Var ( ∑ X i )
By independence:
Var ( X 1 + ⋯ + X n ) = n σ 2 \text{Var}(X_1 + \cdots + X_n) = n\sigma^2 Var ( X 1 + ⋯ + X n ) = n σ 2
For the average X ˉ = 1 n ∑ X i \bar{X} = \frac{1}{n}\sum X_i X ˉ = n 1 ∑ X i
Var ( X ˉ ) = Var ( 1 n ∑ X i ) = 1 n 2 ⋅ n σ 2 = σ 2 n \text{Var}(\bar{X}) = \text{Var}\left(\frac{1}{n}\sum X_i\right) = \frac{1}{n^2} \cdot n\sigma^2 = \frac{\sigma^2}{n} Var ( X ˉ ) = Var ( n 1 ∑ X i ) = n 2 1 ⋅ n σ 2 = n σ 2
Interpretation: Averaging reduces variance. More samples = more precise estimate.
Chebyshev’s Inequality Preview
Variance bounds how often X X X μ \mu μ
P ( ∣ X − μ ∣ ≥ k σ ) ≤ 1 k 2 P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2} P ( ∣ X − μ ∣ ≥ k σ ) ≤ k 2 1
At most 1 / 9 1/9 1/9
Summary
Quantity Formula Variance Var ( X ) = E [ ( X − μ ) 2 ] \text{Var}(X) = E[(X - \mu)^2] Var ( X ) = E [( X − μ ) 2 ] Computational Var ( X ) = E [ X 2 ] − E [ X ] 2 \text{Var}(X) = E[X^2] - E[X]^2 Var ( X ) = E [ X 2 ] − E [ X ] 2 Standard deviation σ X = Var ( X ) \sigma_X = \sqrt{\text{Var}(X)} σ X = Var ( X ) Scaling Var ( a X ) = a 2 Var ( X ) \text{Var}(aX) = a^2 \text{Var}(X) Var ( a X ) = a 2 Var ( X ) Shift invariant Var ( X + c ) = Var ( X ) \text{Var}(X + c) = \text{Var}(X) Var ( X + c ) = Var ( X ) Independent sum Var ( X + Y ) = Var ( X ) + Var ( Y ) \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) Var ( X + Y ) = Var ( X ) + Var ( Y )