Bayes' Theorem and Independence

Bayes’ Theorem Review

$P(F \mid E) = \frac{P(E \mid F) \cdot P(F)}{P(E)}$

This lets us reverse the direction of conditioning: from $P(E \mid F)$ to $P(F \mid E)$.

Computational Form

With partition $F_1, \ldots, F_n$:

$P(F_i \mid E) = \frac{P(E \mid F_i) \cdot P(F_i)}{\sum_{j=1}^{n} P(E \mid F_j) \cdot P(F_j)}$

The denominator is the law of total probability applied to $P(E)$.

Example: Spam Filtering

An email contains the word “free”. Historical data shows:

• $P(\text{"free"} \mid \text{spam}) = 0.3$
• $P(\text{"free"} \mid \text{not spam}) = 0.05$
• $P(\text{spam}) = 0.4$

$P(\text{spam} \mid \text{"free"}) = \frac{0.3 \times 0.4}{0.3 \times 0.4 + 0.05 \times 0.6} = \frac{0.12}{0.12 + 0.03} = \frac{0.12}{0.15} = 0.8$

An email with “free” has 80% probability of being spam.

Independence of Events

Definition

Events $E$ and $F$ are independent if:

$P(E \cap F) = P(E) \cdot P(F)$

Equivalently (when $P(F) > 0$):

$P(E \mid F) = P(E)$

Knowing $F$ occurred gives no information about $E$.

Intuition

Two events are independent if the occurrence of one tells us nothing about the occurrence of the other. The probability of $E$ is the same whether or not we know $F$ occurred.

Example: Coin Flips

Flip a fair coin twice. Let $E$ = “first is heads”, $F$ = “second is heads”.

$P(E \cap F) = P(HH) = \frac{1}{4} = \frac{1}{2} \times \frac{1}{2} = P(E) \cdot P(F)$

The outcomes are independent.

Example: Dependent Events

Draw two cards without replacement. Let $E$ = “first is ace”, $F$ = “second is ace”.

$P(E \cap F) = \frac{4}{52} \cdot \frac{3}{51} = \frac{12}{2652} \neq \frac{4}{52} \cdot \frac{4}{52}$

Not independent: drawing an ace first changes the probability of drawing an ace second.

Independent vs Mutually Exclusive

Common confusion: Independence and mutual exclusivity are different concepts.

Property	$E, F$ disjoint (ME)	$E, F$ independent
Defining equation	$E \cap F = \emptyset$	$P(E \cap F) = P(E) P(F)$
Probability	$P(E \cap F) = 0$	$P(E \cap F) = P(E) P(F) > 0$ typically
Knowing $F$ occurred	$E$ cannot occur	Gives no info about $E$

Key insight: If $E$ and $F$ are both possible (positive probability) and mutually exclusive, they are NOT independent. Knowing $F$ occurred tells us $E$ did NOT occur.

Mutual Independence

Pairwise Independence

Events $E_1, E_2, \ldots, E_n$ are pairwise independent if $E_i$ and $E_j$ are independent for all $i \neq j$.

This is weaker than mutual independence.

Mutual Independence

Events $E_1, E_2, \ldots, E_n$ are mutually independent if for every subset $S \subseteq \{1, 2, \ldots, n\}$:

$P\left(\bigcap_{i \in S} E_i\right) = \prod_{i \in S} P(E_i)$

Important: Pairwise independence does NOT imply mutual independence.

Example: Pairwise but Not Mutual

Flip two fair coins. Define:

• $A$ = first coin is heads
• $B$ = second coin is heads
• $C$ = coins match (both heads or both tails)

Check independence:

• $P(A) = P(B) = P(C) = \frac{1}{2}$
• $P(A \cap B) = \frac{1}{4} = P(A) P(B)$ (pairwise independent)
• $P(A \cap C) = \frac{1}{4} = P(A) P(C)$ (pairwise independent)
• $P(B \cap C) = \frac{1}{4} = P(B) P(C)$ (pairwise independent)

But:

• $P(A \cap B \cap C) = \frac{1}{4}$ (only HH)
• $P(A) P(B) P(C) = \frac{1}{8}$

These events are pairwise independent but NOT mutually independent.

Conditional Independence

Events $E$ and $F$ are conditionally independent given $G$ if:

$P(E \cap F \mid G) = P(E \mid G) \cdot P(F \mid G)$

Events can be independent unconditionally but dependent conditionally, or vice versa.

Example: Berger’s Bullets

Two shooters aim at a target. Let $A, B$ be “shooter A/B hits”. Unconditionally, $A$ and $B$ might be independent. But given the target is hit, knowing $A$ missed makes it more likely $B$ hit, creating dependence.

Properties of Independence

1. If $E$ and $F$ are independent, then so are $E$ and $F^c$.

   Proof: $P(E) = P(E \cap F) + P(E \cap F^c) = P(E) P(F) + P(E \cap F^c)$, so $P(E \cap F^c) = P(E) - P(E) P(F) = P(E)(1 - P(F)) = P(E) P(F^c)$.

2. If $E$ and $F$ are independent, then $E^c$ and $F^c$ are independent.

3. Independence is symmetric: $E$ independent of $F$ iff $F$ independent of $E$.

Application: System Reliability

Components in series all must work. If each has reliability $p$ and failures are independent:

$P(\text{system works}) = p^n$

Components in parallel: at least one must work:

$P(\text{system works}) = 1 - (1-p)^n$

Summary

Concept	Definition
Independence	$P(E \cap F) = P(E) P(F)$
Equivalent form	$P(E \mid F) = P(E)$ (when $P(F) > 0$)
Mutual independence	$P(\bigcap_{i \in S} E_i) = \prod_{i \in S} P(E_i)$ for all subsets $S$
Conditional independence	$P(E \cap F \mid G) = P(E \mid G) P(F \mid G)$