Handling Partial Annotation

The Problem

In real annotation projects, annotators often cover different subsets of items. For example, Annotator D labels items 3-8 whilst Annotator E labels items 1-8. A naive approach is to merge D and E into one “fake” annotator and treat the combined labels as if from a single rater. This creates several problems.

Why Merging D and E Is Wrong

1. Different conditions. D’s decisions on items 3-8 were made at a different time and possibly under different instructions or conditions than E’s decisions on items 1-8. Merging conflates these.

2. Arbitrary bias from discarding. For items both annotated (3-8), you would need to throw away one annotation to avoid duplication. The choice of which to discard introduces arbitrary bias into the dataset.

3. Inconsistent coverage. The “merged” annotator has an inconsistent coverage pattern: some items have one label, some have two. This makes comparisons across items unfair.

4. Asymmetric weighting. If D covered more items than E, the “merged” annotator’s label distribution is skewed toward D’s tendencies. The merged pseudo-annotator is not a real rater; it is a statistical artefact.

The Correct Approach

Adapt Fleiss’ $\kappa$ (or Cohen’s $\kappa$) to handle variable annotator coverage:

1. Calculate $P_a$ only over overlapping items. An item contributes to $P_a$ only if it was labelled by at least two of the full annotator set. For items labelled by only one annotator, no agreement can be computed; they are excluded from the numerator.

2. Calculate $P_e$ from all available annotations. The marginal proportions $p_j$ (how often each class is chosen) are computed across the full dataset, including items with only one label. This gives unbiased estimates of each class’s overall frequency.

3. Use per-item $k$. The denominator $k(k-1)$ for each item uses the number of annotators who actually labelled that specific item, not a global $k$. An item labelled by 2 annotators uses $2 \times 1 = 2$; an item labelled by 5 annotators uses $5 \times 4 = 20$.

Example

Consider a dataset with 5 annotators {A, B, C, D, E} and 8 items. A, B, and C label all 8 items. D labels items 3-8. E labels items 1-8.

For computing $P_a$:

• Items 3-8 are labelled by all 5 annotators: $P_a$ uses these 6 items with $k=5$.
• Items 1-2 are labelled by A, B, C, E: $P_a$ uses these 2 items with $k=4$.

For computing $P_e$:

• All 8 items contribute to the marginal proportions $p_j$.
• Total annotations = $(3 \times 8) + 6 + 8 = 38$ (not $5 \times 8 = 40$).

The global $P_a$ is the mean of per-item $P_{a,i}$ values across all items that have at least 2 annotators.

Why This Matters

Partial annotation is the norm in practical projects. Crowd workers drop out, annotators have different availability, and budgets constrain coverage. Using the correct per-item $k$ and overlapping-only $P_a$ prevents inflated or deflated agreement estimates that would misrepresent annotation quality.

Summary

Problem	Correct solution
Merging annotators conflates different conditions	Keep annotators separate; adapt the formula
Items with single labels have no agreement	Exclude from $P_a$, include in $P_e$
Variable annotator count per item	Use per-item $k$ in the denominator

Past paper questions: y2023p3q7(c)