The Law of Total Probability and Bayes' Theorem

Law of Total Probability

Statement

Let $F_1, F_2, \ldots, F_n$ be a partition of $\Omega$: mutually exclusive events that cover all possibilities. For any event $E$:

$P(E) = \sum_{i=1}^{n} P(E \mid F_i) \cdot P(F_i)$

Intuition

To find the probability of $E$, condition on which scenario $F_i$ occurs and sum over all scenarios, weighting by the probability of each scenario.

Proof

$E = E \cap \Omega = E \cap \bigcup_{i=1}^{n} F_i = \bigcup_{i=1}^{n} (E \cap F_i)$

Since $F_i$ are disjoint, $E \cap F_i$ are also disjoint. By additivity:

$P(E) = \sum_{i=1}^{n} P(E \cap F_i) = \sum_{i=1}^{n} P(E \mid F_i) P(F_i)$

Example: Two Dice

Roll two dice. Let $E$ = “sum is even”. Partition by first die being odd ($F_1$) or even ($F_2$).

• $P(F_1) = P(F_2) = \frac{1}{2}$
• $P(E \mid F_1)$ = first die odd, sum even $\Rightarrow$ second die odd: $\frac{1}{2}$
• $P(E \mid F_2)$ = first die even, sum even $\Rightarrow$ second die even: $\frac{1}{2}$

$P(E) = P(E \mid F_1) P(F_1) + P(E \mid F_2) P(F_2) = \frac{1}{2} \cdot \frac{1}{2} + \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{2}$

Bayes’ Theorem

Statement

$P(F \mid E) = \frac{P(E \mid F) \cdot P(F)}{P(E)}$

Or, using the law of total probability for $P(E)$:

$P(F \mid E) = \frac{P(E \mid F) \cdot P(F)}{\sum_i P(E \mid F_i) \cdot P(F_i)}$

Terminology

• Prior $P(F)$: probability before observing evidence $E$
• Posterior $P(F \mid E)$: updated probability after observing evidence $E$
• Likelihood $P(E \mid F)$: how likely is the evidence under hypothesis $F$
• Marginal likelihood $P(E)$: normalising constant

Proof

By definition of conditional probability:

$P(F \mid E) = \frac{P(F \cap E)}{P(E)} = \frac{P(E \cap F)}{P(E)} = \frac{P(E \mid F) \cdot P(F)}{P(E)}$

Example: Medical Diagnosis

A disease affects 1% of the population. A test has:

• 99% sensitivity: $P(\text{positive} \mid \text{disease}) = 0.99$
• 95% specificity: $P(\text{negative} \mid \text{no disease}) = 0.95$

What is $P(\text{disease} \mid \text{positive})$?

Let $D$ = disease, $T$ = positive test.

$P(D \mid T) = \frac{P(T \mid D) P(D)}{P(T \mid D) P(D) + P(T \mid D^c) P(D^c)} = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.05 \times 0.99} = \frac{0.0099}{0.0594} \approx 16.7\%$

Interpretation: Even with a positive test, there’s only a 17% chance of having the disease (for rare diseases, most positives are false positives).

Example: Monty Hall Problem

You’re on a game show with 3 doors. One hides a prize. You pick door 1. The host opens door 2, revealing no prize. Should you switch to door 3?

Let $F_i$ = prize behind door $i$. Initially $P(F_1) = P(F_2) = P(F_3) = \frac{1}{3}$.

Let $E$ = host opens door 2.

Compute $P(F_1 \mid E)$ and $P(F_3 \mid E)$:

• If $F_1$ (prize behind door 1), host randomly opens door 2 or 3: $P(E \mid F_1) = \frac{1}{2}$
• If $F_2$ (prize behind door 2), host cannot open door 2: $P(E \mid F_2) = 0$
• If $F_3$ (prize behind door 3), host must open door 2: $P(E \mid F_3) = 1$

$P(F_1 \mid E) = \frac{\frac{1}{2} \times \frac{1}{3}}{\frac{1}{2} \times \frac{1}{3} + 0 + 1 \times \frac{1}{3}} = \frac{\frac{1}{6}}{\frac{1}{6} + \frac{1}{3}} = \frac{1}{3}$

$P(F_3 \mid E) = \frac{1 \times \frac{1}{3}}{\frac{1}{2}} = \frac{2}{3}$

Conclusion: Switch to door 3 for a $\frac{2}{3}$ chance of winning.

Example: Quality Control

Two factories produce light bulbs. Factory A supplies 60% of the market with 2% defect rate. Factory B supplies 40% with 5% defect rate. A bulb is defective. What is the probability it came from A?

Let $A$ = from factory A, $B$ = from factory B, $D$ = defective.

$P(A \mid D) = \frac{0.02 \times 0.60}{0.02 \times 0.60 + 0.05 \times 0.40} = \frac{0.012}{0.012 + 0.020} = \frac{0.012}{0.032} = \frac{3}{8}$

Sequential Bayesian Updating

As more evidence arrives, today’s posterior becomes tomorrow’s prior. This is the foundation of Bayesian inference.

Example: Disease Testing Repeat

If we get two independent positive tests:

$P(D \mid T_1 \cap T_2) \propto P(T_1 \cap T_2 \mid D) P(D) = P(T_1 \mid D) P(T_2 \mid D) P(D)$

Starting from prior $P(D) = 0.01$, after one positive test the posterior is $\approx 0.167$. This becomes the new prior for the second test.

Summary

Concept	Formula
Total probability	$P(E) = \sum_i P(E \mid F_i) P(F_i)$
Bayes’ theorem	$P(F \mid E) = \frac{P(E \mid F) P(F)}{P(E)}$
Prior	$P(F)$ before evidence
Posterior	$P(F \mid E)$ after evidence
Likelihood	$P(E \mid F)$ how likely evidence under hypothesis