Evaluation Frameworks, RCTs, and Validity

The Two Core Distinctions

Evaluation methods can be classified along three independent axes. Knowing which quadrant a method occupies is essential for choosing between methods and comparing them in exam answers.

Formative vs Summative

	Formative	Summative
When	Early in the UCD cycle, while designs are still being shaped	Late, when the design is (near) final
Purpose	Compare alternatives, identify weaknesses, guide improvement	Test against predefined, closed criteria; measure whether targets were met
Question type	Open: “what problems exist?”	Closed: “does this design meet the 95% task-completion-rate target?”

Analytical vs Empirical

	Analytical	Empirical
Basis	Applying a theory or framework to the design	Observing and measuring real users interacting with the design
Requires users?	No — the evaluator alone can apply the framework	Yes — real or representative users must be observed
Examples	Heuristic Evaluation, Cognitive Walkthrough, Cognitive Dimensions, KLM/GOMS	Think-aloud protocols, interviews, field observation, controlled experiments, A/B testing

Quantitative vs Qualitative

	Quantitative	Qualitative
Data type	Numbers — enabling statistical comparison	Words, images, audio/video — requiring coding or thematic analysis
Analysis	Statistical tests (t-test, sign test, ANOVA)	Thematic analysis, grounded theory, affinity diagramming

Cross-Tabulation Example

This gives a useful mental map:

• CW and CDs are qualitative-analytical.
• KLM is quantitative-analytical.
• Think-aloud protocols, interviews, and field observation are qualitative-empirical.
• A/B tests and lab trials are quantitative-empirical.

Randomised Controlled Trials (RCTs) and A/B Testing

An RCT in HCI requires:

1. A performance measure — time on task, error rate, task completion rate, or a standardised instrument (e.g. SUS — System Usability Scale).
2. A representative sample of the target population who have given informed consent.
3. An experimental task that yields comparable performance data.

A/B testing is the online, large-scale variant: different users are randomly assigned to different interface versions (A and B), and their behaviour (click-through rate, conversion, dwell time) is compared.

Statistical Testing

The null hypothesis $H_0$: any observed difference between conditions is due to random variation, not the design change.

• Parametric tests (e.g. t-test) assume normally-distributed data — an assumption that often does not hold for small HCI samples.
• Non-parametric alternatives (e.g. the sign test) make fewer assumptions: for each participant in a within-subjects design, record whether the new design was faster (+) or slower (−). Then test whether the proportion of + signs departs from 50/50 by more than would be expected by chance.

Always report confidence intervals / error bars alongside point estimates of the effect size. A statistically significant result is not automatically an interesting one — very small effects can be “significant” with enough data. HCI research typically prioritises large effect sizes over bare statistical significance.

Internal vs External Validity

Type	Question	Threats
Internal validity	”Was the study done right?” — covers reproducibility, scientific integrity, and refutability (could the hypothesis have failed the test?).	Confounding variables (e.g. the new design also used a larger font — was the improvement from the design or the font?). Experimenter bias. Inconsistent procedure across participants.
External validity	”Does the study tell us something useful about the real world?”	Unrepresentative sample (all participants are university students — do findings generalise to older users?). Artificial task (lab-based, no time pressure — does performance predict real-world use?).
Ecological validity (sub-case of external)	“Did the study setting resemble real conditions?”	Lab-based observation of a mobile app while the participant sits at a desk, undistracted — does this predict use while walking, one-handed, in sunlight?

The Hawthorne Effect

The Hawthorne Effect (named after 1920s factory-lighting studies at the Hawthorne Works) occurs when participants change their behaviour simply because they know they are being observed — not because of the experimental manipulation. In the original studies, productivity rose whether lighting was increased or decreased, because workers responded to the attention, not the lighting.

In HCI, this threatens any evaluation where the experimenter is visibly present:

• A participant may persist longer with a confusing interface because they feel observed and do not want to “give up.”
• A participant may rate a design more favourably because they know the experimenter is its designer.
• The “new and interesting” version may receive a temporary enthusiasm boost unrelated to genuine usability improvement.

Mitigation: remote, unmoderated testing; between-subjects designs where possible; standardised, validated instruments (e.g. SUS) rather than ad-hoc rating scales.

Expected Learning

• Distinguish formative from summative, analytical from empirical, and quantitative from qualitative evaluation, classifying given methods correctly.
• Describe what an RCT requires (performance measure, representative sample, informed consent, experimental task).
• Explain the sign test as a non-parametric alternative to the t-test for within-subjects HCI experiments.
• Distinguish internal validity from external validity (including ecological validity as a sub-case).
• Explain the Hawthorne Effect and how it threatens the validity of usability evaluations.
• Understand that statistical significance ≠ practical importance — report effect sizes.

---

Past Paper Questions

RCTs, validity, and the Hawthorne Effect are assessed in: 2020 Q5 (evaluating quantitative findings; sample generalisation), 2019 Q5 (card sorting and similarity analysis — a quantitative-empirical task). The Supervision 2 material includes designing a usability testing research plan with these concepts.