Tokens, Types, and Lexicon-Based Classification

Tokens vs Types

Given the sentence “The cat sat on the mat.”:

• Token: each occurrence of a word. There are 6 tokens: “The”, “cat”, “sat”, “on”, “the”, “mat”. Count includes repetitions.
• Type: each unique word in the vocabulary. There are 5 types: {“the”, “cat”, “sat”, “on”, “mat”}. “The” and “the” are the same type after lowercasing.

Token count $N$ = total words in corpus. Type count $\lvert V \rvert$ = vocabulary size.

Lexicon-Based Classification

A lexicon is a dictionary mapping words to polarities. A simple sentiment lexicon:

Word	Polarity
good	+1
great	+2
terrible	-2
awful	-3

To classify a document: sum the polarities of all lexicon words found. If the sum is positive, classify as positive; negative sum means negative; zero means neutral.

Weaknesses

1. Ignores context. “This film is not good” sums to +1 (from “good”), classified as positive. The negation is missed entirely, because the lexicon contains only unigrams.

2. Cannot handle unseen words. If the document contains “This film is abysmal” and “abysmal” is not in the lexicon, it contributes zero. The document is classified as neutral, missing clear negative sentiment.

3. Fixed vocabulary. Domain shifts break lexicon approaches. A general sentiment lexicon may not capture domain-specific language (e.g., “sick” in gaming communities means “impressive”, not “ill”).

4. No weighting by importance. All lexicon words contribute equally regardless of their discriminating power. “Good” is a weak signal; “masterpiece” is strong. A lexicon with magnitudes (+1, +2, -1, -2) partially addresses this but requires manual calibration.

Lexicon vs Naive Bayes

Lexicon-based classification works when:

• Very little labelled training data is available
• Explicit, inspectable rules are needed
• The domain is well covered by existing lexicons

Naive Bayes works when:

• Sufficient labelled training data exists
• Statistical patterns capture subtle distinctions
• Domain language differs from standard lexicons

The two can be combined: replace words with generic <POSITIVE_WORD> / <NEGATIVE_WORD> tokens before training Naive Bayes. This bridges the lexicon’s generalisation with the classifier’s ability to learn from data.

Summary

Concept	Definition	Example
Token	Each word occurrence	”the cat the cat” = 4 tokens
Type	Each unique word	”the cat the cat” = 2 types
Lexicon-based classification	Sum word polarities	Fast, interpretable, context-blind
Lexicon + NB combination	Replace words with polarity tokens	Generalises across synonyms

Past paper questions: y2021p3q9(e)(ii-iii), y2025p3q7(a)