Correlation

Definition

The correlation coefficient (Pearson correlation) is the normalised covariance:

$\rho_{X,Y} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$

where $\sigma_X = \sqrt{\text{Var}(X)}$ and $\sigma_Y = \sqrt{\text{Var}(Y)}$.

Properties

1. Unitless

Correlation has no units, making it comparable across different measurements.

2. Range

$-1 \leq \rho_{X,Y} \leq 1$

Proof (Cauchy-Schwarz): Consider $E[(a(X-E[X]) + b(Y-E[Y]))^2] \geq 0$.

3. Interpretation

$\rho$	Relationship
$\rho = 1$	Perfect positive linear relationship
$\rho = -1$	Perfect negative linear relationship
$\rho = 0$	No linear relationship
$\rho > 0$	Positive linear association
$\rho < 0$	Negative linear association

4. Linear Transformation Invariant

$\rho_{aX+b, cY+d} = \text{sign}(ac) \cdot \rho_{X,Y}$

Correlation doesn’t change if we scale or shift $X$ and $Y$.

Visual Interpretation

• $\rho \approx 1$: points cluster along a line with positive slope
• $\rho \approx -1$: points cluster along a line with negative slope
• $\rho \approx 0$: no linear pattern

Proof of Range

By Cauchy-Schwarz for random variables:

$E[UV]^2 \leq E[U^2] E[V^2]$

Let $U = X - E[X]$ and $V = Y - E[Y]$:

$\text{Cov}(X, Y)^2 \leq \text{Var}(X) \cdot \text{Var}(Y)$

$\rho^2 = \frac{\text{Cov}(X,Y)^2}{\text{Var}(X)\text{Var}(Y)} \leq 1$

Thus $|\rho| \leq 1$.

When is $\rho = \pm 1$?

$\rho = 1$ if and only if $Y = aX + b$ for some $a > 0$ (perfect positive linear).

$\rho = -1$ if and only if $Y = aX + b$ for some $a < 0$ (perfect negative linear).

Independence vs Zero Correlation

• Independent $\Rightarrow \rho = 0$
• $\rho = 0$ does NOT $\Rightarrow$ independent

Counterexample

Let $X$ be uniform on $\{-1, 0, 1\}$, and $Y = X^2$.

$\text{Cov}(X, Y) = E[X^3] - E[X]E[X^2] = 0 - 0 = 0$

$\rho = 0$, but $X$ and $Y$ are clearly dependent (knowing $X$ determines $Y$).

The issue: correlation measures linear dependence only. $Y = X^2$ is a perfect nonlinear relationship.

Sample Correlation

From data $(X_1, Y_1), \ldots, (X_n, Y_n)$:

$r = \frac{\sum(X_i - \bar{X})(Y_i - \bar{Y})}{\sqrt{\sum(X_i - \bar{X})^2} \sqrt{\sum(Y_i - \bar{Y})^2}}$

Example

From earlier: drawing cards, $\text{Cov}(X, Y) = -\frac{1}{272}$.

$\text{Var}(X) = \frac{1}{4} \cdot \frac{3}{4} = \frac{3}{16}$, same for $\text{Var}(Y)$.

$\rho = \frac{-1/272}{\sqrt{3/16} \cdot \sqrt{3/16}} = \frac{-1/272}{3/16} = -\frac{16}{272 \cdot 3} = -\frac{1}{51}$

Small negative correlation.

Summary

Property	Value
Definition	$\rho = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$
Range	$[-1, 1]$
$\rho = 1$	Perfect positive linear
$\rho = -1$	Perfect negative linear
$\rho = 0$	No linear relationship
Limitation	Doesn’t capture nonlinear dependence