Naive Bayes versus Lexicon-Based Approaches

Two Approaches to Text Classification

Naive Bayes is a supervised statistical classifier: it learns $P(w \mid c)$ from labelled training data by counting word-class co-occurrences. The decision rule is:

$c^* = \arg\max_{c} \; P(c) \prod_{i=1}^{n} P(w_i \mid c)$

Lexicon-based classification uses a hand-built dictionary where each word has a pre-assigned polarity or category (e.g., “terrible” → negative, “great” → positive). The decision is typically a sum or majority vote over lexicon matches. No training data is needed.

When Naive Bayes Wins

Naive Bayes is the better choice when:

• Sufficient labelled training data exists. The classifier needs enough examples per class to estimate reliable word probabilities. With thousands of labelled messages, NB can detect subtle statistical patterns.
• Statistical patterns capture subtle discrimination. Combinations of seemingly neutral words (e.g., “you should know your place”) can together signal bullying even though no individual word is in a bullying lexicon. NB captures these co-occurrence patterns.
• Domain language differs from standard lexicons. A general sentiment lexicon performs poorly on specialised domains like gaming slang, medical reports, or legal documents. NB learns the domain-specific vocabulary directly from the labelled data.

When Lexicon-Based Wins

A lexicon is better when:

• Very little labelled data is available. Building a domain lexicon requires modest expert effort but no labelled examples.
• The domain is well-covered by existing lexicons. Sentiment analysis with standard dictionaries (LIWC, VADER, SentiWordNet) needs no training data.
• You need explicit, inspectable rules. A human can read the lexicon and understand exactly which words trigger which decisions. This is essential in regulated settings where decisions must be explainable and auditable.

The Best-of-Both Strategy

Use lexicon tokens as Naive Bayes features: replace specific words with generic category tokens before training.

Original text	After lexicon preprocessing
”you are awful and ugly"	"you are <NEGATIVE> and <NEGATIVE>"
"that is brilliant work"	"that is <POSITIVE> work”

This hybrid approach provides four advantages:

• Handles unseen synonyms: if “grotesque” is in the lexicon but never appeared in training, it maps to <NEGATIVE> and NB can use it immediately.
• Handles language change: new bullying slang (e.g., “ratio’d”, “mid”) can be added to the lexicon without retraining the full NB model.
• Reduces vocabulary size dramatically: all positive words collapse into one feature, all negative words into another. The vocabulary $|V|$ shrinks, reducing the zero-probability problem and improving smoothing.
• Combines strengths: NB captures word co-occurrence and overall message tone; the lexicon captures expert knowledge about specific words and handles the long tail.

Cyberbullying Detection Example (2021 Q9)

In cyberbullying detection, language change is the critical threat. Younger users constantly invent new words and repurpose existing ones. A fixed NB classifier degrades rapidly.

Building a Cyberbullying Lexicon

Method 1 (new words): track unknown words during classification. If a word suddenly spikes in frequency, sample messages containing it, have them manually labelled, and add the word to the lexicon if bullying-related. The labelled messages become new training data.

Method 2 (new senses): find “maximally surprising” messages where the classifier was confident but wrong. Check whether known words in these messages have acquired a new threatening sense. Add these to the lexicon and use the surprising messages for retraining.

Robustness to Language Change

Run the lexicon-based detector on new messages to cheaply identify likely bullying instances. Use these as new training material for periodic NB retraining. The lexicon acts as a lightweight, continuously-updated scout. Active learning approaches can also help: find the messages where the classifier is most uncertain, have humans label those, and retrain NB.

Evaluation with Lexicons

When a company policy demands “delete as many bullying messages as possible while keeping non-filtered messages high,” this translates to: optimise recall (find all bullying) while maintaining reasonable precision (don’t falsely flag OK messages). Precision and recall — not accuracy — are the right metrics because bullying messages are rare, and accuracy would be dominated by correctly identifying the abundant OK messages. A development corpus helps find the probability threshold that balances precision and recall for the new policy.

Comparison Summary

Criterion	Naive Bayes	Lexicon
Requires labelled data	Yes, substantial	No
Handles unseen words	Via smoothing	If word is in lexicon
Handles language change	Poorly (needs retraining)	Well (update lexicon)
Captures subtle patterns	Yes (co-occurrence)	No (word-level only)
Interpretability	Inspectable parameters	Explicit rules
Domain adaptation	Learns from data	Requires domain lexicon
Best-of-both	Lexicon tokens as NB features	Bridges both worlds

Past paper questions: y2021p3q9, y2025p3q7