Intrinsic Evaluation: WCSS and Silhouette

What Is Intrinsic Evaluation?

Intrinsic evaluation requires no ground truth labels. It assesses cluster quality based solely on properties of the data and the clustering itself: how tight are the clusters? How well separated are they?

Two main intrinsic metrics: WCSS and Silhouette.

WCSS (Within-Cluster Sum of Squares)

Formula

$\text{WCSS} = \sum_{n=1}^{N} \lVert x_n - \mu_{z_n} \rVert^2$

where $\mu_{z_n}$ is the centroid of the cluster $x_n$ is assigned to (identified by $z_n$).

Interpretation

Lower WCSS means tighter, more compact clusters. A single cluster containing all points has high WCSS. Many singleton clusters (one point each) have WCSS = 0.

The Elbow Method

WCSS always decreases as $K$ increases (more clusters = tighter fit). The elbow method plots WCSS against $K$ and chooses the $K$ at the “elbow”: the point where the rate of decrease flattens sharply.

Procedure:

1. Run K-Means for $K = 1, 2, 3, \ldots, K_{\text{max}}$.
2. Record WCSS for each $K$.
3. Plot WCSS vs $K$.
4. The optimal $K$ is the elbow: the last $K$ before diminishing returns set in.

Limitation: the elbow is often ambiguous for real-world data where WCSS decreases smoothly with no clear knee point.

Silhouette Score

Per-Point Formula

For a point $i$ assigned to cluster $C$:

• $a(i)$ = mean distance from $i$ to all other points in the same cluster $C$ (cohesion).
• $b(i)$ = mean distance from $i$ to all points in the nearest other cluster (separation; the cluster with smallest mean distance to $i$ that is not $C$).

$S(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$

Interpretation

• $S(i)$ near $+1$: point is much closer to its own cluster than to the nearest other cluster (well clustered).
• $S(i)$ near $0$: point lies on the boundary between two clusters.
• $S(i)$ near $-1$: point is closer to a different cluster than its own (likely misclustered).

The overall silhouette score is the mean of $S(i)$ across all points.

Advantages over WCSS

The silhouette score captures both cohesion and separation; WCSS only measures cohesion. Silhouette works for any distance metric, not just Euclidean. It can identify individual misclustered points.

Worked Example: Silhouette for One Point

Three clusters: $C_1 = \{(1,2), (2,1), (2,2)\}$, $C_2 = \{(8,6), (9,7), (10,6)\}$, $C_3 = \{(5,8), (6,9)\}$.

Compute $S(i)$ for point $p = (1,2)$ in $C_1$.

Step 1: $a(p)$ — cohesion within $C_1$

Distances to other points in $C_1$:

• To $(2,1)$: $\sqrt{(1-2)^2 + (2-1)^2} = \sqrt{2} \approx 1.41$
• To $(2,2)$: $\sqrt{(1-2)^2 + (2-2)^2} = 1$

$a(p) = \frac{1.41 + 1.00}{2} = 1.21$

Step 2: $b(p)$ — separation to nearest other cluster

Mean distance to $C_2$:

• To $(8,6), (9,7), (10,6)$: mean $\approx \sqrt{(1-9)^2 + \ldots} = 9.18$

Mean distance to $C_3$:

• To $(5,8), (6,9)$: mean $\approx \sqrt{(1-5)^2 + \ldots} = 8.45$

$C_3$ is nearer, so $b(p) = 8.45$.

Step 3: $S(p)$

$S(p) = \frac{8.45 - 1.21}{8.45} = \frac{7.24}{8.45} \approx 0.86$

The point is well clustered (close to its own cluster, far from others).

Summary

Metric	Formula	Measures	Range	Best when
WCSS	$\sum \lVert x_n - \mu_{z_n} \rVert^2$	Cohesion only	$[0, \infty)$	Low
Silhouette $S(i)$	$(b-a)/\max(a,b)$	Cohesion + separation	$[-1, 1]$	Near +1
Elbow method	Plot WCSS vs $K$	Optimal $K$	—	Clear elbow
$a(i)$	Intra-cluster mean distance	Cohesion	—	Low
$b(i)$	Nearest-cluster mean distance	Separation	—	High

Past paper questions: y2024p3q7e