Cohen's Kappa for Two Annotators

Why Raw Agreement Misleads

If two annotators label items independently, simply counting how often they agree gives a misleading picture. Even if both annotators guess randomly, they will agree by chance some fraction of the time. With few classes (e.g., binary labels), chance agreement can be high: two annotators both guessing “OK” 50% of the time agree by chance 50% of the time.

Cohen’s $\kappa$ corrects for this: it measures agreement above chance, normalised by the maximum possible agreement above chance.

Formula

$\kappa = \frac{P(A) - P(E)}{1 - P(E)}$

• $P(A)$: observed agreement — proportion of items where both annotators gave the same label.
• $P(E)$: expected chance agreement — the probability they would agree if both guessed independently based on their observed label distributions.

$P(E) = \sum_{c \in \text{Classes}} P(\text{Ann}_1 \text{ chooses } c) \times P(\text{Ann}_2 \text{ chooses } c)$

Interpretation

• $\kappa = 1.0$: perfect agreement
• $\kappa > 0.8$: strong agreement
• $\kappa > 0.6$: acceptable agreement
• $\kappa = 0.0$: no better than chance
• $\kappa < 0$: worse than chance (systematic disagreement)

Full Worked Example (2023 Q7)

Two annotators (A and B) classify 8 items into 4 classes (I, II, III, IV). Data:

Item	1	2	3	4	5	6	7	8
A	III	IV	II	I	II	I	IV	II
B	III	I	II	IV	IV	I	IV	I

Step 1 — Observed agreement $P(A)$: Items where A = B: item 1 (III=III), item 3 (II=II), item 6 (I=I), item 7 (IV=IV).

$P(A) = 4/8 = 0.5$

Step 2 — Marginal frequencies for A:

• $P_A(\text{I}) = 2/8 = 0.25$
• $P_A(\text{II}) = 3/8 = 0.375$
• $P_A(\text{III}) = 1/8 = 0.125$
• $P_A(\text{IV}) = 2/8 = 0.25$

Step 3 — Marginal frequencies for B:

• $P_B(\text{I}) = 3/8 = 0.375$
• $P_B(\text{II}) = 1/8 = 0.125$
• $P_B(\text{III}) = 1/8 = 0.125$
• $P_B(\text{IV}) = 3/8 = 0.375$

Step 4 — Expected chance agreement $P(E)$:

$P(E) = (0.25 \times 0.375) + (0.375 \times 0.125) + (0.125 \times 0.125) + (0.25 \times 0.375)$

$P(E) = 0.09375 + 0.046875 + 0.015625 + 0.09375 = 0.25$

Step 5 — Compute $\kappa$:

$\kappa = \frac{0.5 - 0.25}{1 - 0.25} = \frac{0.25}{0.75} = 0.333$

Interpretation: moderate agreement, well above chance ($\kappa = 0.0$) but far from strong ($\kappa > 0.8$). The annotators do better than random guessing but disagree substantially.

Why This Matters for ML

If human annotators achieve only $\kappa = 0.333$ on a task, a classifier trained on their labels cannot realistically exceed that performance. The inter-annotator agreement sets an upper bound on achievable classifier accuracy. Low $\kappa$ indicates the task itself is subjective, and the “ground truth” labels are noisy.

Summary

Component	Meaning	This example
$P(A)$	Observed agreement	0.5 (4 of 8)
$P(E)$	Chance agreement	0.25
$\kappa$	Agreement above chance, normalised	0.333

Past paper questions: y2023p3q7(b), y2021p3q9(c)