Cross-Validation

N-Fold Cross-Validation

When data is limited, a single train-dev-test split wastes precious training examples. N-fold cross-validation (CV) maximises data usage: divide the data into $N$ equal folds, train on $N-1$ folds, test on the remaining fold, and rotate until every fold has served as the test set. Report the mean and variance of the metric across folds.

For $N=5$ and 100 documents: partition into 5 folds of 20 documents each. In each round, train on 80 documents, test on 20. After 5 rounds, average the 5 accuracy scores (or F1 scores).

Five-fold or ten-fold CV is standard. Ten-fold gives lower bias but higher variance; five-fold is computationally cheaper.

Stratified Cross-Validation

Standard CV splits data randomly. With imbalanced classes (e.g., 90% OK messages, 10% bullying), a random split could place all bullying messages in one fold, distorting evaluation. Stratified CV ensures each fold preserves the overall class distribution: if the full dataset is 90/10, every fold is also ~90/10.

Stratified CV is essential for any imbalanced classification task.

Leave-One-Out CV (Jack-Knifing)

The extreme case: $N$ equals the number of data points. For 100 documents, train on 99 and test on 1, repeating 100 times. This maximises training data and gives the least biased estimate of performance. The cost is high: 100 training runs, each on nearly the full dataset. Only practical for very small datasets (tens of items).

Dependency-Sensitive Cross-Validation

When data points are not independent (e.g., multiple documents by the same author, or from the same publication venue), random CV overestimates generalisation. The model memorises author style rather than learning the task.

Dependency-sensitive CV folds by domain: all documents from author X go into a single fold, ensuring the model is tested on genuinely unseen authors, genres, or sources. This tests true out-of-domain generalisation.

Worked Example

A dataset contains 100 documents across 2 classes (60 OK, 40 bullying). With 5-fold stratified CV:

• Each fold holds 20 documents: 12 OK and 8 bullying (preserving the 60/40 ratio).
• In round 1: train on folds 2-5 (80 documents, 48 OK + 32 bullying), test on fold 1.
• After 5 rounds: compute mean accuracy = 0.87, variance = 0.002.
• Report: “87% accuracy (SD = 4.5%) across 5-fold stratified CV.”

Without stratification, a fold might contain only 2 bullying messages, producing unreliable per-fold estimates and inflating variance.

Summary

Method	How it works	When to use
N-fold CV	Train on N-1 folds, test on 1, rotate	Limited data, standard approach
Stratified CV	Preserve class distribution per fold	Imbalanced data
Leave-one-out	N = dataset size	Tiny datasets; computationally expensive
Dependency-sensitive CV	Fold by domain (author, source, genre)	Testing generalisation to new domains

Past paper questions: y2021p3q9 (evaluation strategy)