Detecting and Proving Bias in Naive Bayes

Why Naive Bayes Is Interpretable

Naive Bayes is interpretable: you can inspect every parameter — every $P(w \mid c)$, every $P(c)$ — directly from the probability table. You can trace any classification decision back to which specific features contributed which specific probabilities.

The useful property: identify bias immediately by scanning the parameter table for zeros. If $P(\text{feature} \mid \text{class}) = 0$, that class can never be predicted when that feature is present. You can prove this mathematically with two test instances.

The problematic property: zero probabilities propagate. Multiplying small probabilities together means one zero anywhere in the product makes the entire posterior zero — the classifier becomes completely insensitive to all other evidence. A single unseen word or feature value kills the classifier for that instance.

This interpretability is impossible with black-box models (neural networks, random forests, SVMs). You cannot inspect individual parameters, trace decisions to specific training examples, or construct guaranteed counter-examples.

The 4-Step Method for Proving Bias

To mathematically demonstrate that a classifier is biased against a protected group:

1. Identify a protected attribute (Gender, Ethnicity, Nationality, Age).
2. Find a value of that attribute that only appears in one class in the training data (i.e., $P(\text{value} \mid \text{other\_class}) = 0$).
3. Construct two identical test instances differing ONLY in the protected attribute. All other features should appear in both classes (neutral features).
4. Show that the classifier gives different predictions for the two instances, or show $P(\text{class} \mid \text{protected\_value}) = 0$, meaning that class can NEVER be predicted for any instance with that protected value — regardless of all other features.

Full Worked Example: Football Academy (2024 Q9)

Training data: 6 recruits with features Goals (Many/Few/None), Position (Attack/Defender/Goalkeeper), Gender (M/F), and label Success (Y/N).

Step 1 — Priors

$P(Y) = \frac{3}{6} = 0.5 \qquad P(N) = \frac{3}{6} = 0.5$

Step 2 — Likelihoods (MLE, no smoothing)

Goals:

$P(\text{Many} \mid Y) = \frac{1}{3} \quad P(\text{Few} \mid Y) = \frac{1}{3} \quad P(\text{None} \mid Y) = \frac{1}{3}$

$P(\text{Many} \mid N) = 0 \quad P(\text{Few} \mid N) = \frac{2}{3} \quad P(\text{None} \mid N) = \frac{1}{3}$

Position:

$P(\text{Attack} \mid Y) = \frac{2}{3} \quad P(\text{Defender} \mid Y) = \frac{1}{3} \quad P(\text{Goalkeeper} \mid Y) = 0$

$P(\text{Attack} \mid N) = 0 \quad P(\text{Defender} \mid N) = \frac{2}{3} \quad P(\text{Goalkeeper} \mid N) = \frac{1}{3}$

Gender:

$P(\text{M} \mid Y) = 1 \quad P(\text{F} \mid Y) = 0$ $P(\text{M} \mid N) = \frac{2}{3} \quad P(\text{F} \mid N) = \frac{1}{3}$

Bias Group 1: Goalkeepers (Protected: Position)

Protected value: Position = Goalkeeper. Only appears with class N in training.

Test instance: Goals=None, Position=Goalkeeper, Gender=M.

$P(Y \mid \text{instance}) \propto P(Y) \cdot P(\text{None} \mid Y) \cdot P(\text{Goalkeeper} \mid Y) \cdot P(\text{M} \mid Y)$ $= 0.5 \cdot \frac{1}{3} \cdot 0 \cdot 1 = 0$

A Goalkeeper can NEVER be predicted as Successful — regardless of their Goals scored or Gender. The zero from $P(\text{Goalkeeper} \mid Y)$ propagates through the product and wipes out all other information.

Bias Group 2: Female Players (Protected: Gender)

Protected value: Gender = F. Only appears with class N in training.

Test instance: Goals=Many, Position=Attack, Gender=F.

$P(Y \mid \text{instance}) \propto 0.5 \cdot \frac{1}{3} \cdot \frac{2}{3} \cdot 0 = 0$

A Female player can NEVER be predicted as Successful — regardless of their Goals or Position. Every $P(Y \mid \dots)$ is zero the moment Gender=F appears.

Why This Happened

The training data encoded historical discrimination: no Female players and no Goalkeepers appeared in the Successful group. Naive Bayes faithfully reproduced the patterns in the data. This mirrors the London Medical School Admissions case: a program built to replicate human admissions decisions reproduced the gender and ethnic discrimination present in the historical training data. “Fitting the data” does not produce fair decisions if the data itself reflects historical injustice.

Summary

Concept	Key Point
Interpretability	Every $P(w \mid c)$ is directly inspectable — impossible with black-box models
Zero propagation	One zero in the NB product kills all classification power
4-step proof	Protected attribute → find exclusive value → construct test pair → show inequality
Goalkeeper bias	$P(\text{Goalkeeper} \mid Y) = 0$, can never predict Y
Gender bias	$P(\text{F} \mid Y) = 0$, can never predict Y
Root cause	Historical discrimination encoded in training data; NB faithfully reproduces it

Past paper questions: y2024p3q9