Covariance

Motivation

Variance measures spread. But how do we measure the joint variability of two random variables? Do they tend to move together or in opposite directions?

Definition

The covariance of $X$ and $Y$ is:

$\text{Cov}(X, Y) = E[(X - E[X])(Y - E[Y])]$

Computational Formula

$\text{Cov}(X, Y) = E[XY] - E[X]E[Y]$

Proof:

$E[(X - E[X])(Y - E[Y])] = E[XY - XE[Y] - E[X]Y + E[X]E[Y]]$ $= E[XY] - E[X]E[Y] - E[X]E[Y] + E[X]E[Y] = E[XY] - E[X]E[Y]$

Properties

1. Symmetry

$\text{Cov}(X, Y) = \text{Cov}(Y, X)$

2. Variance is Self-Covariance

$\text{Cov}(X, X) = E[X^2] - E[X]^2 = \text{Var}(X)$

3. Linearity in Each Argument

$\text{Cov}(aX + b, Y) = a \cdot \text{Cov}(X, Y)$

$\text{Cov}(X, Y + Z) = \text{Cov}(X, Y) + \text{Cov}(X, Z)$

4. Variance of Sum

$\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y)$

General:

$\text{Var}\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n \text{Var}(X_i) + 2\sum_{i < j} \text{Cov}(X_i, X_j)$

5. Independent Variables Have Zero Covariance

If $X$ and $Y$ are independent: $\text{Cov}(X, Y) = 0$.

Warning: $\text{Cov}(X, Y) = 0$ does NOT imply independence.

Interpretation

Covariance	Interpretation
$\text{Cov}(X, Y) > 0$	Positive relationship: $X$ and $Y$ tend to move together
$\text{Cov}(X, Y) < 0$	Negative relationship: $X$ and $Y$ tend to move oppositely
$\text{Cov}(X, Y) = 0$	No linear relationship

Example

Roll two dice. Let $X$ = first die, $Y$ = second die, $S = X + Y$.

$E[X \cdot S] = E[X(X+Y)] = E[X^2] + E[XY] = \frac{91}{6} + \frac{49}{4} = \dots$

Since $X$ and $Y$ independent: $\text{Cov}(X, Y) = 0$.

But $\text{Cov}(X, S) = \text{Cov}(X, X + Y) = \text{Var}(X) + \text{Cov}(X, Y) = \frac{35}{12} + 0 = \frac{35}{12} > 0$.

$X$ and $S$ are positively correlated: higher $X$ tends to mean higher sum.

Units

Covariance has units of $X \times Y$. If $X$ is in metres and $Y$ in seconds, $\text{Cov}(X,Y)$ is in metre-seconds.

This makes interpretation difficult. Correlation (below) normalises to be unitless.

Example: Pair from Deck

Draw two cards without replacement. $X$ = number of hearts in first, $Y$ = number of hearts in second.

Both $X$ and $Y$ are Bernoulli with $p = \frac{1}{4}$.

By symmetry: $E[X] = E[Y] = \frac{13}{52} = \frac{1}{4}$.

$E[XY] = P(X = 1, Y = 1) = P(X = 1) P(Y = 1 \mid X = 1) = \frac{13}{52} \cdot \frac{12}{51} = \frac{156}{2652}$

$\text{Cov}(X, Y) = E[XY] - E[X]E[Y] = \frac{156}{2652} - \frac{1}{16} = \frac{1}{17} - \frac{1}{16} = -\frac{1}{272}$

Negative covariance: drawing a heart first reduces the chance of drawing a heart second.

Summary

Property	Formula
Definition	$\text{Cov}(X,Y) = E[(X-E[X])(Y-E[Y])]$
Computational	$\text{Cov}(X,Y) = E[XY] - E[X]E[Y]$
Variance of sum	$\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X,Y)$
Independent	$\Rightarrow \text{Cov}(X,Y) = 0$
Converse	$\text{Cov}(X,Y) = 0 \not\Rightarrow$ independent