Case Study: Pneumonia-Asthma Interpretability
Background (Caruana et al.)
A neural network was trained to predict the risk of complications or death for patients hospitalised with pneumonia. The goal was clinical triage: identify high-risk patients who need intensive care and low-risk patients who could be treated as outpatients.
What the Model Learned
The neural network achieved high accuracy on the test set. However, inspection of its behaviour revealed a disturbing pattern: the model had learned that having asthma was associated with a lower risk of dying from pneumonia. In the model’s internal logic, “asthma = protective factor.”
The Real Reason
The model’s conclusion was factually wrong in the most dangerous possible way. The true causal relationship is the opposite: asthma increases pneumonia mortality risk. The data showed lower mortality for asthma patients only because asthma patients with pneumonia were fast-tracked to intensive care (ICU) by doctors who recognised their vulnerability. The faster, more aggressive treatment saved lives, making the survival rate for asthmatics appear higher in the data.
The model learned a correlation (asthma → lower mortality in the historical data) but not the underlying causation (asthma → fast-tracked to ICU → lower mortality). Deploying this model would mean sending asthmatic pneumonia patients home, where they would die at higher rates.
The Interpretability Contrast
A transparent rule-based system (a decision tree or simple rule list) was also built for the same task. When its rules were inspected, the logical flaw was immediately obvious: a rule saying “if asthma, then low risk” was clearly wrong to any medical professional. The rule was caught before deployment.
The neural network, being uninterpretable, concealed the same flaw. Its high test-set accuracy provided false confidence. The model was dangerously unsuitable for clinical triage, yet its accuracy score gave no warning of this.
The Lesson
Interpretability is a safety requirement in high-stakes domains, not a convenience. In clinical medicine, criminal justice, credit scoring, and autonomous vehicles, “the model is accurate” is insufficient. You must be able to verify that the model has learned the right thing for the right reasons.
High test-set accuracy does not imply the model has learned correct causal relationships. It implies only that it has learned patterns that correlate with the labels in the test distribution. If the test distribution shares the same confounded patterns as the training distribution, accuracy is meaningless for safety.
Connection to the MLRD Syllabus
This case pairs with the bias detection content: just as you can inspect Naive Bayes parameters to find discriminatory zero-probabilities, you can inspect interpretable models to find dangerous causal misunderstandings. The syllabus emphasises that Naive Bayes’ interpretability is a feature, not just an implementation detail.
Summary
| Aspect | Detail |
|---|---|
| Authors | Caruana et al. |
| Model | Neural network for pneumonia triage |
| Flaw | Learned that asthma lowers mortality (wrong; actually opposite) |
| Real reason | Asthmatics were fast-tracked to ICU → higher survival in data |
| Rule-based system | Caught the flaw immediately (transparent) |
| Neural network | Concealed the flaw (black-box) |
| Lesson | Interpretability is a safety requirement, not a convenience |
Past paper questions: y2024p3q9