Algorithmic Policy & Governance · Capstone brief
POL-02 · Disparate impact worksheet
Run a fairness-style audit on a public dataset with a metric set you declare first. Where samples are thin, refuse the claim rather than decorate it with a percentage.
The question
Run a fairness-style audit on a public dataset with a metric set you declare first. Where samples are thin, refuse the claim rather than decorate it with a percentage.
Policy / materials
Teacher-approved public dataset and task. Metric set fixed before analysis (e.g. selection-rate gaps and error-rate gaps — student must define formulas in the worksheet). Threshold for “too thin to claim” written before running numbers.
Expected failure modes
Picking metrics after seeing which group “wins.” Reporting rates for cells of five people. Causal language from observational data. Hiding the thin-cell rule after it triggers.
Done looks like
Worksheet with pre-declared metrics; results table with counts; thin-cell refusals highlighted; one-page interpretation for a non-statistician; explicit non-claim list.
Five C's
CT: counts before narratives. CR: pre-registration of metrics. CO: peer re-computes two cells. CM: one-page brief. CZ: the group described by a fragile rate.
Mentor role
Stats teacher or social-science teacher reviews thin-cell rule before analysis. Standing instruction: reject post-hoc metric shopping. School-supervised.
Rubric calibration
R1: one dataset, one decision. R2: reproducible worksheet. R3: baseline is overall rate. R4: thin cells refused. R5: plain-language page. R6: non-claim list.
Two ways this goes wrong
(a) A heatmap with no counts. (b) “Bias proven” from a cell of eight.
Checkpoint suggestions
- Week 1–2: Dataset, decision, metrics, and thin-cell rule approved.
- Week 4–5: Tables complete; peer recompute; refusals marked.
- Week 7–8: One-page brief; mentor review.
Credit lane fit
Lane A immediately. No verified credit claim.