Crowdsourcing Problems and Pitfalls
Why Crowdsourcing Annotations Is Difficult
Crowdsourcing platforms distribute annotation tasks to many workers, each labelling a small batch of items. Whilst this is cheap and fast, it introduces several problems that make standard inter-annotator agreement metrics unreliable or impossible to compute.
Problem 1: Very Small Batches
When each worker annotates only 2-3 items, the variance in per-worker agreement estimates is enormous. A worker who happens to get two easy items looks excellent; a worker who gets two ambiguous items looks terrible. Neither estimate is reliable. With large batches (20+ items per worker), you can form stable per-worker quality estimates; with 2 items, you cannot.
Problem 2: Random Assignment Breaks Pairwise Kappa
Crowd workers are typically assigned random subsets of items. With hundreds of workers each labelling a few items, many annotator pairs never share a single item. Cohen’s requires pairs to have at least some overlapping items. Pairwise agreement between workers who never labelled the same thing cannot be computed.
Fleiss’ partially solves this (it pools all annotators), but still assumes most items have multiple labels, which may not hold if workers are spread thinly.
Problem 3: Variable Worker Quality
Crowd workers vary widely in skill and motivation. Some are diligent experts; others spam randomly to maximise payment per unit time. A worker who labels everything “POS” will agree with other workers purely by chance on balanced datasets. Simple agreement metrics cannot distinguish genuine skill from lucky guessing or systematic spamming.
Solutions: include gold-standard items (with known labels) to filter low-quality workers; use worker-specific quality weights in the agreement calculation; reject workers whose accuracy on gold items falls below a threshold.
Problem 4: No Inter-Section Consistency Check
Large annotation projects often divide items into sections, with different worker pools per section. If no workers label items across sections, there is no way to check whether labelling standards are consistent between sections. Section A’s annotators might interpret the guidelines differently from Section B’s. The resulting dataset has systematic bias between sections, but no agreement metric will reveal it because there is no overlap to measure against.
Solution: allocate a shared subset of items across all sections. These overlap items serve as a consistency check, allowing computation of agreement between sections.
Summary
| Problem | Cause | Consequence |
|---|---|---|
| Small batches | 2 items per worker | High variance, unreliable quality estimates |
| Random assignment | No shared items between pairs | Pairwise impossible |
| Variable quality | Some workers spam randomly | Raw agreement inflates with lucky guesses |
| No inter-section overlap | Sections use separate worker pools | Hidden systematic bias between sections |
Past paper questions: y2023p3q7(c)(ii-iii)