Clinical AI & Diagnostic Bias · Capstone brief
CLIN-08 · Risk scores vs. clinical judgment
Where does the score add information, where does it merely add steps, and at what decision threshold does the answer change?
The question
Compare a published clinical risk score against a deliberately simple rule — one or two variables a nurse could apply from the doorway. Where does the score add information, where does it merely add steps, and at what decision threshold does the answer change?
System / materials
One published risk score with public coefficients or a documented point system, plus either a public dataset to apply both to, or the score's own published performance figures for a metrics-only comparison. The simple rule is written by the student before seeing results and approved by the teacher. Useful frame: decision curve analysis, which asks whether a model helps at the thresholds anyone would actually act on — Vickers & Elkin, Medical Decision Making, 2006 (https://pubmed.ncbi.nlm.nih.gov/17099194/).
Expected failure modes
Choosing a straw-man rule so the score wins. Comparing on discrimination alone when the clinical question is a decision at a threshold. Ignoring the cost of collecting the score's inputs — a model needing four labs is not free at triage. Declaring the simple rule superior on a sample too small to separate them.
Done looks like
A comparison memo: the simple rule stated first and timestamped as pre-specified; both approaches evaluated at a named decision threshold with error counts as people; an input-cost accounting (what must be measured, by whom, how long); and a recommendation naming the setting where each wins — including the honest possibility that they are indistinguishable in the available evidence.
Five C's
CT: separating discrimination from decision usefulness. CR: writing a fair simple rule in advance. CO: a peer applies the simple rule to ten cases and agreement is reported. CM: a memo a charge nurse would read. CZ: who is missed under each approach.
Mentor role
A clinician or nurse educator reviews the simple rule before results are computed. Standing instruction: reject a rule designed to lose. School-supervised.
Rubric calibration
R1: one score, one rule, one threshold. R2: both reproducible from cited sources. R3: the simple rule is the baseline and is pre-specified. R4: input cost and sample limits both stated. R5: recommendation is actionable. R6: refuses a winner the evidence cannot support.
Two ways this goes wrong
(a) AUCs compared, no threshold, no decision, no nurse. (b) The simple rule is invented after seeing the score's weak spots.
Credit lane fit
Lane A immediately (health-science capstone, HOSA, or nursing-pathway seminar). No verified credit claim.