The Multiple Comparisons Problem

The Problem

Repeatedly evaluating on the test set during development makes the test set no longer “unseen”. Each evaluation is a comparison; across many comparisons, the model that happens to look best on this particular random split is selected. Statistical significance tests assume a single, pre-planned comparison against fresh data. Using the same test set across iterations violates this assumption.

Why p-Values Break

A p-value of 0.03 means: “if the null hypothesis were true, there is a 3% chance of observing a result this extreme.” This relies on the test data being a single fresh draw. If you have tested 50 model variants on the same test set, you have effectively run 50 comparisons. The probability that at least one variant looks “significant” at $\alpha = 0.05$ purely by chance is:

$P(\text{at least one significant}) = 1 - (1 - 0.05)^{50} \approx 0.923$

There is a 92% chance that pure noise will produce at least one “significant” result. The p-value of the winning variant is meaningless, because that variant won precisely because it overfit the noise in the test set.

Common Exam Scenario

A student splits data into train/test, then develops a classifier over 15 iterations:

1. Train on training set
2. Evaluate on test set
3. Adjust features based on test-set performance
4. Repeat

On iteration 15, accuracy = 88%. The student computes a p-value and claims the classifier is significantly better than a baseline. This is invalid. The model was selected from 15 candidates based on test-set performance. The test set has been incorporated into the model selection process. The 88% is optimistically biased, and the associated p-value is incorrect.

The Correct Approach

Split data into three portions: train, dev, test. Use the dev set for all iterative development (feature selection, hyperparameter tuning, model selection). Only when the final model is locked down, evaluate once on the test set and report that single number as the honest performance estimate. The test set is touched exactly once.

Connection to the Sign Test

The same logic applies to significance testing between systems. If you run a sign test on the test set, then tweak the system and run another sign test on the same test set, you are performing multiple comparisons. The second test’s p-value is invalid unless corrected. The test set must be held out from all development and only used for one pre-planned significance test.

Summary

Issue	Cause	Consequence
Multiple comparisons	Repeated test-set evaluation during development	p-values become invalid
Optimistic bias	Selecting model that looks best on test split	Reported performance is inflated
Correct fix	Use separate dev set for iteration; test set once	Valid statistical inference

Past paper questions: y2021p3q9 (evaluation strategy)