Clinical AI & Diagnostic Bias · Capstone brief
CLIN-05 · Label noise in clinical data
On a public clinical dataset, define two defensible labeling rules for the same outcome — a stricter and a looser one — and measure how much your conclusions move when only the definition changes. Does the finding survive the definition?
The question
The label is not the disease. On a public clinical dataset, define two defensible labeling rules for the same outcome — a stricter and a looser one — and measure how much your conclusions move when only the definition changes. Does the finding survive the definition?
System / materials
One public clinical dataset with enough documentation to see how the outcome was derived — coded diagnosis, lab threshold, procedure occurrence, or chart-derived flag. Small tabular sets are ideal (https://archive.ics.uci.edu/); PhysioNet resources may work if the access tier is open and confirmed first (https://physionet.org/). The teacher approves both labeling rules in writing before any model is fit. No PHI, and no relabeling of anything a student saw on placement.
Expected failure modes
Choosing a second rule that is obviously worse so the first one wins. Reporting only the metric change and not the change in who gets labeled positive. Tuning the model between the two runs, which confounds the whole comparison. Concluding the data is "noisy" without naming which decision introduced the noise.
Done looks like
A definition-sensitivity memo: both labeling rules written as reproducible criteria with clinical justification; the same fixed model and split run under each; a comparison of headline metrics and of the labeled cohorts themselves — how many patients switch class, and whether the switchers share a characteristic; and a stated conclusion about which findings held under both definitions and which did not.
Five C's
CT: treating the outcome definition as a modeling choice. CR: writing a second rule in good faith. CO: a peer applies the written rules to ten records and gets the same labels, or the rules get rewritten. CM: a memo that leads with what changed, not with the pipeline. CZ: who the switchers are, and what happens to a patient on the wrong side of a definition.
Mentor role
A clinician, coder, or clinical informaticist reviews both labeling rules before modeling. Standing instruction: reject a straw-man second rule. School-supervised.
Rubric calibration
R1: one outcome, two written rules, one fixed model. R2: rules reproducible by another student without asking. R3: comparator is the same model under the original label. R4: cohort shift reported as people, not only as metric delta. R5: conclusion states what survived. R6: refuses to report the better-looking rule as the true one.
Two ways this goes wrong
(a) Two AUCs, one sentence, no examination of who moved. (b) The model is retuned under the second label, so nothing about the comparison is interpretable.
Credit lane fit
Lane A immediately (health-science capstone or AP Research). Needs a student comfortable with a basic modeling workflow. No verified credit claim.