Type 1 and Type 2 Errors, Power, and Specificity

The Four Outcomes

Every significance test yields one of four outcomes, depending on the true state of the world and the test’s conclusion:

	H0 is TRUE	H0 is FALSE
Reject H0	Type 1 Error ($\alpha$)	Correct rejection (Power, $1-\beta$)
Do NOT reject H0	Correct non-rejection (Specificity, $1-\alpha$)	Type 2 Error ($\beta$)

Type 1 Error (False Positive, $\alpha$)

Rejecting H0 when it is actually true: claiming System A is better when it is not. The significance level $\alpha = 0.05$ means we accept a 5% chance of committing a Type 1 error every time we run a test.

If we run 20 independent tests at $\alpha = 0.05$, the expected number of Type 1 errors is 1. This is the multiple comparisons problem: across many tests, some will appear significant purely by chance.

Type 2 Error (False Negative, $\beta$)

Failing to reject H0 when it is actually false: missing a real improvement. $\beta$ is the probability of a Type 2 error. Unlike $\alpha$ (which we set explicitly), $\beta$ depends on the true effect size, sample size, and test power. A small sample makes $\beta$ large: real differences go undetected.

Power ($1 - \beta$)

The probability of correctly rejecting a false H0: detecting a real difference when one exists. Higher power is better. Power increases with:

• Larger sample size $N$
• Larger true effect size
• More sensitive test (permutation test is more powerful than sign test)

A test with low power cannot reliably detect real improvements. If you fail to reject H0 with a small dataset, you cannot conclude “A and B are the same”; you can only conclude “the data are insufficient to tell.”

Specificity ($1 - \alpha$)

The probability of correctly not rejecting a true H0: guarding against false discoveries. If $\alpha = 0.05$, specificity = 0.95. Raising the significance threshold (e.g., $\alpha = 0.01$) increases specificity but reduces power.

The Trade-Off

$\alpha$ and $\beta$ are in tension. Lowering $\alpha$ (making it harder to reject H0) reduces Type 1 errors but increases Type 2 errors (since you need stronger evidence to reject). Raising $\alpha$ increases power but also increases false positives.

The standard $\alpha = 0.05$ balances Type 1 and Type 2 errors for most applications. In safety-critical domains (medical treatment), a lower $\alpha$ (more conservative) may be warranted to avoid false positives. In exploratory research, a higher $\alpha$ may be acceptable to avoid missing genuine signals.

Worked Example

A sign test on 200 documents compares two spam filters. H0: filters are equally good. The observed test statistic gives $p = 0.03$. We reject H0 at $\alpha = 0.05$ and conclude Filter A is significantly better.

If H0 were actually true (filters are identical), we have committed a Type 1 error: the significant result was a fluke. There is a 5% chance of this occurring, and it did.

If H0 were actually false (Filter A is genuinely better) and we had instead obtained $p = 0.07$, we would fail to reject H0: a Type 2 error. The test lacked power to detect a real difference.

Summary

Concept	Symbol	Meaning
Type 1 Error	$\alpha$	False positive: claiming A is better when it is not
Type 2 Error	$\beta$	False negative: missing a real improvement
Power	$1-\beta$	Correctly detecting a real improvement
Specificity	$1-\alpha$	Correctly avoiding false discoveries

Past paper questions: y2023p3q8 (evaluation)