Independence of Random Variables

Definition

Random variables $X$ and $Y$ are independent if:

$F(x, y) = F_X(x) \cdot F_Y(y) \quad \text{for all } x, y$

Equivalently (for the usual cases):

Discrete: $p(x, y) = p_X(x) \cdot p_Y(y) \quad \text{for all } x, y$

Continuous: $f(x, y) = f_X(x) \cdot f_Y(y) \quad \text{for all } x, y$

Intuition

$X$ and $Y$ are independent if knowing $X$ gives no information about $Y$ (and vice versa). The joint distribution factors into the product of marginals.

Checking Independence

Method 1: Factor Joint into Marginals

Compute $p_X(x)$ and $p_Y(y)$, then verify $p(x, y) = p_X(x) p_Y(y)$ everywhere.

Example: $p(x, y) = e^{-x-y}$ for $x, y > 0$.

$p_X(x) = e^{-x}$ (marginal) $p_Y(y) = e^{-y}$ (marginal)

$p_X(x) p_Y(y) = e^{-x} e^{-y} = e^{-x-y} = p(x, y)$

Independent.

Method 2: Check if Support is Rectangular

If the support of $(X, Y)$ is not a rectangle (or hyper-rectangle for $n > 2$), then $X$ and $Y$ cannot be independent.

Example: $f(x, y) = 6xy$ on $0 \leq x \leq 1$, $0 \leq y \leq \sqrt{x}$.

The support is triangular, not rectangular, so $X$ and $Y$ are NOT independent.

Independence Implies

If $X$ and $Y$ are independent:

1. Joint Probability Factors

$P(X \in A, Y \in B) = P(X \in A) \cdot P(Y \in B)$

2. Expectation of Product

$E[XY] = E[X] E[Y]$

Proof: $E[XY] = \sum_x \sum_y xy \cdot p(x,y) = \sum_x \sum_y xy \cdot p_X(x) p_Y(y) = \left(\sum_x x p_X(x)\right)\left(\sum_y y p_Y(y)\right) = E[X]E[Y]$

3. Variance of Sum

$\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$

This follows from $E[XY] = E[X]E[Y]$ for independent variables.

4. Functions of Independent Variables

If $X$ and $Y$ are independent, then $g(X)$ and $h(Y)$ are also independent for any functions $g, h$.

Independence Does NOT Imply

Common misconceptions:

• Zero correlation $\neq$ independence: Independent implies zero correlation, but the converse is false
• Uncorrelated $\neq$ independent: Zero covariance doesn’t guarantee independence

Counterexample

Let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$.

$\text{Cov}(X, Y) = E[X \cdot X^2] - E[X] E[X^2] = E[X^3] - 0 = 0$

But $X$ and $Y$ are clearly not independent: knowing $X$ completely determines $Y$.

Estimates from Data

If we observe $(X_i, Y_i)$ pairs, we estimate:

• Marginal of $X$: distribution of $X_i$ values alone
• Marginal of $Y$: distribution of $Y_i$ values alone
• Independence: check if joint empirical distribution factorises

Mutual Independence

$X_1, X_2, \ldots, X_n$ are mutually independent if:

$F(x_1, \ldots, x_n) = F_{X_1}(x_1) \cdots F_{X_n}(x_n)$

For all subsets, the joint factors into the product of marginals.

Pairwise independence does NOT imply mutual independence.

Summary

Condition	Statement
Independence	Joint = product of marginals
Implies	$E[XY] = E[X]E[Y]$
Implies	$\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$
Zero correlation	Does NOT imply independence