Analysing Qualitative Data

Once data has been gathered — interview transcripts, observation notes, diary entries, open-ended questionnaire responses — it must be systematically analysed to extract findings that can inform design. Raw data is not evidence; analysed data is.

Categorical Coding (for Closed Questions)

When the research questions are defined in advance and the data fits into pre-existing categories:

1. Build a coding frame of expected categories before looking at the data. The categories should be mutually exclusive (no overlap) and collectively exhaustive (every data point fits somewhere).
2. Segment the text — break transcripts into analysable units (a phrase, a sentence, a short paragraph expressing one idea).
3. Assign each segment to one category.
4. Compare frequency and co-occurrence of categories across groups (e.g. do novice users mention “confusion” more often than experts?).

Inter-rater reliability must be measured and reported: have at least two independent coders code the same data, then compute agreement. Cohen’s $\kappa$ for two coders; Fleiss’ $\kappa$ for more than two. $\kappa \in [0.6, 0.8]$ is typically considered “substantial agreement”; below 0.4 is “poor.” Reporting $\kappa$ protects against the criticism that the coding frame is merely the researcher’s subjective interpretation.

Grounded Theory (for Open Questions, No Prior Hypothesis)

Grounded theory is used when you do not know in advance what categories will emerge — the theory is built from the ground up from the data:

1. Open coding: read the data closely line by line, tagging every potentially interesting concept without a predetermined frame.
2. Write memos capturing insights, connections, and emerging patterns as you code.
3. Axial coding: organise the open codes into broader themes, identifying relationships between categories.
4. Constant comparison: continuously compare new data against the emerging themes and memos — do the themes still hold? Do they need revising?
5. Stop at theoretical saturation: when new data no longer changes the themes or generates new insights, the theory is stable. Continuing to collect data beyond saturation wastes resources without improving understanding.

Thematic Analysis

A lighter-weight relative of grounded theory. Instead of building a full theory, the researcher identifies recurring patterns (themes) across a dataset. Thematic analysis is more descriptive and less theoretically ambitious than grounded theory, but it is faster and more practical for many applied design-research contexts.

Affinity Diagramming

A practical, collaborative technique used in design workshops rather than formal research. Individual findings (observations, quotes, pain points) are written on sticky notes. The design team physically groups related notes on a wall or whiteboard, creating clusters that reveal emergent themes. Unlike grounded theory, affinity diagramming is fast, collaborative, and can be done in a single afternoon — but it lacks the rigour, inter-rater reliability, and saturation criterion of formal qualitative analysis.

Validity and Reliability in Qualitative Research

Concept	How to improve it
Validity (are you measuring what you claim to measure?)	Triangulation: use multiple methods or data sources (e.g. interview + observation + questionnaire) to cross-check findings. Member-checking: show your findings back to participants and ask “does this reflect your experience?”
Reliability (would another researcher get the same results?)	Use multiple independent coders and report inter-rater agreement ($\kappa$). Document your coding process so it is auditable.

Bad Evaluation Techniques — Never Do This

Some common but indefensible practices to recognise and avoid:

Leading Questions

“Do you like this nice new interface more than that ugly old one?”

Looks quantitative (a 1–10 rating) but is biased by experimental demand — the participant feels pressure to agree with the researcher, especially if they are a friend or colleague. The wording embeds the desired answer.

Unevidenced Assertions

“It was deemed that more colours should be used to increase usability.”

Phrased to sound like a research finding, but is pure opinion — no evidence, no method, no participants. The passive voice (“it was deemed”) hides the absence of any actual evaluation.

The HiPPO Method

“I find it more intuitive this way, and the users will too.”

HiPPO = Highest-Paid Person’s Opinion. Common in industry, where a senior stakeholder’s personal preference overrides research evidence. The designer (or CEO, or product manager) is not a representative user — their technical knowledge, domain expertise, and familiarity with the design make them the worst possible judge of its usability for actual users. Never present HiPPO as evidence in an exam answer or in professional practice.

Expected Learning

• Describe the four steps of categorical coding and explain why inter-rater reliability ($\kappa$) must be reported.
• Contrast grounded theory with thematic analysis and affinity diagramming in terms of rigour, time, and appropriate use contexts.
• Define triangulation and member-checking, and explain how each improves validity.
• Identify and critique bad evaluation techniques (leading questions, unevidenced assertions, HiPPO) in a given scenario.
• Explain theoretical saturation and why it matters for grounded theory.

---

Past Paper Questions

Qualitative data analysis is discussed in: 2024 Q5 (ChatGPT — interview-data analysis), 2023 Q6 (mindfulness app — interpreting qualitative user research). The supervision material directly covers coding and analysis methods.