Train, Development, and Test Splits
The Golden Rule
Before performing any analysis on a dataset, partition it into three disjoint subsets: training, development (validation), and test. This separation prevents data leakage and preserves the validity of statistical evaluation.
Training Set
The training set is used strictly for fitting model parameters. For a Naive Bayes classifier, this means counting word-class co-occurrences and estimating priors. The model sees these examples during learning; no evaluation happens here.
Typical proportion: 60-80% of the total data.
Development (Validation) Set
The development set tunes hyperparameters, selects features, and debugs the model. You may evaluate on this set repeatedly whilst iterating on the model: adjusting the smoothing parameter , choosing which features to include, comparing different model architectures.
Typical proportion: 10-20% of the total data.
Test Set
The test set is used exactly once: to report the final, unbiased performance of the chosen model. It must remain completely unseen throughout development. The model’s parameters must never be influenced by test-set performance.
Typical proportion: 10-20% of the total data.
The Multiple Comparisons Problem
If you check the test set repeatedly during development (even “just to see how things are going”), you are implicitly tuning to it. Each peek constitutes a comparison; across many peeks, the model that happens to score highest on this particular test split will be selected. The test set is no longer “unseen”, and statistical significance tests become invalid.
A common exam scenario: a student develops a model over 20 iterations, checking test-set accuracy after each change. On the final iteration, they obtain 85% and claim significance. This is wrong: the model was selected precisely because it happened to look best on that test split. The p-value associated with that 85% is meaningless, since the test data has been incorporated into the model selection process.
Simple Diagram
A typical data split maps as follows: 70% of the original data becomes the training set. Of the remaining 30%, half (15% of total) forms the development set and half (15% of total) forms the held-out test set. The training set fits the model; the development set guides feature and hyperparameter selection; the test set provides one final, honest evaluation number.
Summary
| Split | Purpose | Touched |
|---|---|---|
| Training | Fit parameters (counts, priors) | Every training run |
| Development | Tune hyperparameters, select features | Repeatedly during development |
| Test | Final unbiased evaluation | ONCE only |
Past paper questions: y2021p3q9, y2023p3q8