Probability Theory

Basic Definitions

Sample space $\Omega$: set of all possible outcomes.

Event: subset of $\Omega$.

Probability: $P(A) \in [0, 1]$ for event $A$. Properties:

• $P(\Omega) = 1$
• $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ for disjoint events: $P(A \cup B) = P(A) + P(B)$

---

Conditional Probability

$P(A|B) = \frac{P(A \cap B)}{P(B)}$

“Probability of $A$ given that $B$ has occurred.”

Bayes’ theorem:

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

Independence: $A$ and $B$ are independent if $P(A \cap B) = P(A)P(B)$.

---

Random Variables

A random variable $X$ is a function from $\Omega$ to $\mathbb{R}$.

Discrete: Takes countably many values.

Continuous: Takes values in an interval or $\mathbb{R}$.

---

Probability Distributions

Discrete: Probability Mass Function (PMF)

$p(x_i) = P(X = x_i)$

Requirements: $p(x_i) \geq 0$ and $\sum_i p(x_i) = 1$.

Continuous: Probability Density Function (PDF)

$f(x)$ such that:

$P(a \leq X \leq b) = \int_a^b f(x)\,dx$

Requirements: $f(x) \geq 0$ and $\int_{-\infty}^{\infty} f(x)\,dx = 1$.

---

Expectation Value (Mean)

Discrete: $E[X] = \mu = \sum_i x_i p(x_i)$

Continuous: $E[X] = \mu = \int_{-\infty}^{\infty} x f(x)\,dx$

Linearity: $E[aX + bY] = aE[X] + bE[Y]$

---

Variance and Standard Deviation

Variance: $\text{Var}(X) = \sigma^2 = E[(X - \mu)^2] = E[X^2] - (E[X])^2$

Standard deviation: $\sigma = \sqrt{\text{Var}(X)}$

Properties:

• $\text{Var}(aX) = a^2\text{Var}(X)$
• $\text{Var}(X + c) = \text{Var}(X)$

---

Moments

The $n$th moment is $E[X^n]$.

The $n$th central moment is $E[(X - \mu)^n]$.

• 1st moment = mean
• 2nd central moment = variance
• 3rd central moment (scaled) = skewness
• 4th central moment (scaled) = kurtosis

---

Binomial Distribution

$n$ independent trials, each with success probability $p$.

$P(X = r) = \binom{n}{r} p^r (1-p)^{n-r}$

where $\binom{n}{r} = \frac{n!}{r!(n-r)!}$.

Mean: $E[X] = np$

Variance: $\text{Var}(X) = np(1-p)$

---

Poisson Distribution

For rare events with average rate $\lambda$:

$P(X = r) = \frac{\lambda^r e^{-\lambda}}{r!}$

Mean: $E[X] = \lambda$

Variance: $\text{Var}(X) = \lambda$

Limit: Poisson is the limit of Binomial as $n \to \infty$, $p \to 0$ with $np = \lambda$ fixed.

---

Normal (Gaussian) Distribution

$f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$

Notation: $X \sim N(\mu, \sigma^2)$.

Mean: $\mu$

Variance: $\sigma^2$

Standard normal ($\mu = 0$, $\sigma = 1$):

$f(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}$

Cumulative distribution function: $\Phi(z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-t^2/2}\,dt$

---

Central Limit Theorem

If $X_1, X_2, \ldots, X_n$ are independent, identically distributed random variables with mean $\mu$ and variance $\sigma^2$, then:

$\frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \to N(0, 1) \quad \text{as } n \to \infty$

where $\bar{X} = \frac{1}{n}\sum X_i$.

Significance: Sample means tend toward normal distribution, regardless of the underlying distribution (provided it has finite variance).

---

Stirling’s Approximation

$\ln(n!) \approx n\ln n - n$

More precisely: $n! \approx n^n e^{-n} \sqrt{2\pi n}$

Derivation sketch: $\ln(n!) = \sum_{k=1}^{n} \ln k \approx \int_1^n \ln x\,dx = n\ln n - n + 1$

Use in probability: Essential for approximating binomial coefficients in limiting cases.