The Confusion Matrix and Evaluation Metrics

The 2×2 Confusion Matrix

For binary classification, every prediction falls into one of four cells:

	Truth: Positive	Truth: Negative
System says: Positive	TP (True Positive)	FP (False Positive)
System says: Negative	FN (False Negative)	TN (True Negative)

• TP: correctly identified positives (bullies caught)
• FP: false alarms (innocent messages flagged)
• FN: missed positives (bullies that slipped through)
• TN: correctly identified negatives (innocent messages passed)

Accuracy

$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{FP} + \text{FN} + \text{TN}}$

Accuracy is misleading when classes are imbalanced. A classifier that labels every message “OK” on a dataset with 95% OK messages achieves 95% accuracy, yet detects zero bullying. Always check class distribution before trusting accuracy.

Precision

$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$

Of all messages the system claimed were bullying, what fraction truly were? High precision means when the system flags something, you can trust it.

Prioritise high precision when: medical treatment decisions, legal evidence flagging, any domain where false alarms carry high cost.

Recall

$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

Of all actual bullying messages, what fraction did the system find? High recall means few bullies are missed.

Prioritise high recall when: disease screening, spam detection, brand-attack detection, any domain where missing a positive is worse than a false alarm.

F1 Score

$\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

The harmonic mean of precision and recall. F1 penalises extreme imbalance: if precision = 0.9 and recall = 0.1, F1 = 0.18 (not 0.5). This makes F1 more honest than the arithmetic mean.

Prioritise high F1 when: most NLP classification tasks, balanced trade-off needed.

Worked Example

A bullying-filter classifier is evaluated on 1000 messages, of which 100 are truly bullying:

	Truth: Bullying	Truth: OK
System says: Bullying	TP = 80	FP = 40
System says: OK	FN = 20	TN = 860

• Accuracy = (80 + 860) / 1000 = 0.94 (94%)
• Precision = 80 / (80 + 40) = 80/120 = 0.667
• Recall = 80 / (80 + 20) = 80/100 = 0.80
• F1 = 2 × (0.667 × 0.80) / (0.667 + 0.80) = 2 × 0.534 / 1.467 = 0.728

Accuracy looks excellent at 94%, but that is driven by the 860 TNs. Precision and recall reveal that one third of flagged messages are false alarms, and one fifth of bullies are missed.

Note: precision and recall are always defined with respect to a specific class (here, bullying). There is no such thing as “overall precision” without specifying the class of interest.

Summary

Metric	Formula	Use
Accuracy	(TP+TN)/Total	Balanced classes only
Precision	TP/(TP+FP)	When false alarms are costly
Recall	TP/(TP+FN)	When missing positives is costly
F1	2PR/(P+R)	Balanced trade-off

Past paper questions: y2021p3q9(b)