Feature Engineering and Generalisation
Why Feature Engineering Matters
Raw text produces a sparse, high-dimensional feature space. Naive Bayes’ independence assumption becomes less tenable, and zero-probability events proliferate. Feature engineering reduces sparsity and improves generalisation to unseen data.
Lowercasing
Convert all text to lowercase: “Running” and “running” become the same type.
Why it helps: Reduces vocabulary size without losing discriminative information. Capitalisation rarely carries class-relevant signal in standard text classification. A halved vocabulary means fewer unseen-word events and more reliable probability estimates per type.
Stemming and Lemmatisation
Map morphological variants to a common root:
- “argued”, “arguing”, “argues”, “argument” → “argu” (stemming)
- “better”, “good” → “good” (lemmatisation, handles irregular forms)
Why it helps: Groups related forms together, increasing the count for each root and reducing vocabulary size. The probability estimate for “argu” is based on the combined evidence from all its surface forms, making it more reliable.
Stop-Word Removal
Remove high-frequency function words: “the”, “and”, “is”, “of”, “to”, “a”.
Why it helps: These words appear in every document regardless of class. They carry no discriminative power and add noise. Removing them reduces document length and vocabulary size without affecting the classifier’s ability to separate classes.
Hapax Legomena Removal
Remove words that occur exactly once in the training data (hapax legomena, singular: hapax legomenon).
Why it helps: A word seen only once provides an unreliable probability estimate. It may be a typo, a rare proper noun, or statistical noise. Removing hapaxes reduces vocabulary size and prevents the classifier from overfitting to idiosyncratic training tokens that will never appear again. The trade-off: some rare but genuinely informative words are lost, but the noise reduction usually outweighs this.
Binning Continuous Features
For numerical features (age, income, scores), group values into discrete brackets:
- Age 18, 19, 20 → [17-20]
- Score 72, 88, 91 → [70-80], [80-90], [90-100]
Why it helps: Naive Bayes requires discrete features. A continuous feature like age has potentially infinite values, each with zero or negligible counts. Binning reduces the feature to a small set of categories, each with meaningful probability estimates. The bracket width can be tuned on the development set.
Lexicon Integration
Replace specific words with generic sentiment tokens:
- “awful”, “terrible”, “dreadful” →
<NEGATIVE_WORD> - “excellent”, “wonderful”, “brilliant” →
<POSITIVE_WORD>
Why it helps: Unseen synonyms of known sentiment words are mapped to the same token, so the classifier recognises their polarity even without having seen the specific word in training. This handles language change and domain shift: a new slang term for “bad” is still replaced by <NEGATIVE_WORD> if it appears in the lexicon.
Combined Approach
These techniques are not mutually exclusive. A typical pipeline: lowercase → remove stop words → stem → remove hapaxes → replace lexicon words with polarity tokens. The development set guides which combination produces the best held-out performance.
Summary
| Technique | What it does | Why it helps |
|---|---|---|
| Lowercasing | ”Running” → “running” | Halves vocabulary, same discriminative power |
| Stemming/lemmatisation | ”argued”/“arguing” → “argu” | Groups variants, more reliable counts |
| Stop-word removal | Drop “the”, “and”, “is” | Removes non-discriminative noise |
| Hapax removal | Drop words seen once | Removes unreliable probability estimates |
| Binning | Age 18 → [17-20] | Reduces sparsity for numerical features |
| Lexicon integration | ”awful” → <NEGATIVE_WORD> | Handles unseen synonyms, language change |
Past paper questions: y2021p3q9(c)(ii), y2021p3q9(e)(iii), y2024p3q7(d)