Fleiss' Kappa for Multiple Annotators

When Cohen’s Kappa Is Not Enough

Cohen’s $\kappa$ works for exactly two annotators. When you have three or more annotators (e.g., a crowd of workers each labelling items), you need Fleiss’ $\kappa$, which generalises the chance-corrected agreement measure to an arbitrary number of raters.

Notation

• $N$: number of items
• $k$: number of annotators per item (can vary per item in the generalised version)
• $C$: number of categories (classes)
• $n_{ij}$: number of annotators who assigned category $j$ to item $i$

Formula

Observed agreement per item:

$P_{a,i} = \frac{\sum_{j=1}^{C} n_{ij}(n_{ij} - 1)}{k(k - 1)}$

The numerator counts agreeing pairs for item $i$ (summed over categories). The denominator is the total number of annotator pairs for that item. The factor of 2 in the combination formula cancels between numerator and denominator.

Overall observed agreement:

$P_a = \frac{1}{N} \sum_{i=1}^{N} P_{a,i}$

Expected chance agreement:

For each category $j$, compute the overall proportion of all annotations assigned to that class:

$p_j = \frac{\sum_{i=1}^{N} n_{ij}}{N \times k}$

Then:

$P_e = \sum_{j=1}^{C} p_j^2$

Fleiss’ $\kappa$:

$\kappa = \frac{P_a - P_e}{1 - P_e}$

Interpretation follows the same scale as Cohen’s $\kappa$.

Full Worked Example

Three annotators (A, B, C) label 3 items as POS or NEG:

Item	Annotator A	Annotator B	Annotator C
1	POS	POS	NEG
2	POS	NEG	NEG
3	POS	POS	POS

Here $N=3$, $k=3$, $C=2$.

Step 1 — $n_{ij}$ counts:

Item	$n_{\text{POS}}$	$n_{\text{NEG}}$
1	2	1
2	1	2
3	3	0

Step 2 — $P_{a,i}$ per item:

Item 1: $P_{a,1} = [2(2-1) + 1(1-1)] / [3(3-1)] = [2 + 0] / 6 = 0.333$

Item 2: $P_{a,2} = [1(1-1) + 2(2-1)] / 6 = [0 + 2] / 6 = 0.333$

Item 3: $P_{a,3} = [3(3-1) + 0(0-1)] / 6 = [6 + 0] / 6 = 1.0$

Step 3 — Overall $P_a$:

$P_a = (0.333 + 0.333 + 1.0) / 3 = 1.667 / 3 = 0.556$

Step 4 — Category proportions $p_j$:

Total annotations = $3 \times 3 = 9$. POS annotations = $2 + 1 + 3 = 6$, so $p_{\text{POS}} = 6/9 = 0.667$. NEG annotations = $1 + 2 + 0 = 3$, so $p_{\text{NEG}} = 3/9 = 0.333$.

Step 5 — Expected chance agreement $P_e$:

$P_e = 0.667^2 + 0.333^2 = 0.444 + 0.111 = 0.556$

Step 6 — Fleiss’ $\kappa$:

$\kappa = \frac{0.556 - 0.556}{1 - 0.556} = \frac{0}{0.444} = 0.0$

Interpretation: agreement is no better than chance. Despite $P_a = 0.556$ looking modestly above 50%, the chance correction reveals that the observed agreement is exactly what random guessing with the observed label proportions would produce.

Summary

Component	Meaning	This example
$P_a$	Mean pairwise agreement per item	0.556
$P_e$	Chance agreement from marginal proportions	0.556
$\kappa$	Fleiss’ kappa	0.0 (chance-level)

Past paper questions: y2023p3q7(b), y2021p3q9(c)