Introduction to Estimators

From Theory to Data

Probability: given distribution, compute probabilities.

Statistics: given data, infer distribution.

Estimation is statistical inference: using data to estimate unknown parameters.

Estimators

An estimator $\hat{\theta}$ is a function of the data used to estimate an unknown parameter $\theta$.

The estimate is the value computed from actual data.

Example: Estimating Population Mean

Parameter: $\mu$ (unknown population mean)

Data: $X_1, \ldots, X_n$ (random sample)

Estimator: $\hat{\mu} = \bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$

Estimate: 42.3 (actual value from data)

Properties of Good Estimators

Unbiasedness

$\hat{\theta}$ is unbiased if $E[\hat{\theta}] = \theta$.

The expected value of the estimator equals the true parameter.

Low Variance

$\text{Var}(\hat{\theta})$ should be small.

An estimator with low variance concentrates near its mean.

Consistency

$\hat{\theta}$ is consistent if $\hat{\theta} \xrightarrow{P} \theta$ as $n \to \infty$.

With more data, the estimator converges to the truth.

Example: Sample Mean

$\bar{X}$ as estimator for $\mu$:

Property	Status
Unbiased	Yes: $E[\bar{X}] = \mu$
Variance	$\sigma^2/n$
Consistent	Yes (by WLLN)

Example: Sample Variance

Two versions:

$S_n^2 = \frac{1}{n}\sum_{i=1}^n(X_i - \bar{X})^2 \quad \text{(biased)}$

$S^2 = \frac{1}{n-1}\sum_{i=1}^n(X_i - \bar{X})^2 \quad \text{(unbiased)}$

Dividing by $n-1$ corrects for estimating $\mu$ with $\bar{X}$.

Physical Analogy

Estimator = Measuring instrument Parameter = True value Estimate = Measurement Sampling variation = Measurement error

$\text{Measurement} = \text{True value} + \text{Error}$

Point vs Interval Estimation

• Point estimate: Single value (e.g., $\bar{X} = 42.3$)
• Interval estimate: Range (e.g., 95% CI: $(40.1, 44.5)$)

Interval estimates convey uncertainty.

Method of Moments

Match sample moments to population moments:

$E[X] = \bar{X}$ $E[X^2] = \frac{1}{n}\sum X_i^2$

Solve for parameters.

Maximum Likelihood

Find parameter value maximising probability of observed data.

$\hat{\theta}_{MLE} = \arg\max_\theta L(\theta) = \arg\max_\theta \prod_{i=1}^n f(X_i; \theta)$

Summary

Concept	Definition
Estimator	Function of data to estimate parameter
Estimate	Value computed from actual data
Unbiased	$E[\hat{\theta}] = \theta$
Consistent	$\hat{\theta} \to \theta$ as $n \to \infty$