Applied ML Systems · Capstone brief
DS-25 · Break a published result
Choose a published model or benchmark result on a classic public dataset. Reproduce it. Then find the conditions under which it fails, and write the model card its authors did not.
The question
Choose a published model or benchmark result on a classic public dataset. Reproduce it. Then find the conditions under which it fails, and write the model card its authors did not.
Data
UCI Machine Learning Repository (https://archive.ics.uci.edu/). Optionally a Hugging Face dataset with a published baseline (https://huggingface.co/datasets). The chosen paper or model card must be publicly readable without a paywall.
Expected failure modes
Reproduction usually fails first for boring reasons — preprocessing, split seeds, library versions — and students conclude the result was wrong. Several beloved UCI datasets have known documented problems, which is a feature here. "Finding failure" can slide into cherry-picking a hostile subset.
Done looks like
A documented reproduction with any gap from the published number explained, a systematic robustness probe (distribution shift, subgroup performance, label noise, or feature ablation), and a model card with intended use, out-of-scope use, and measured limitations.
Five C's
CT: distinguishing a reproduction failure from a genuine refutation. CR: designing probes the original authors did not run. CO: swapping papers with a peer and attempting each other's reproduction. CM: writing a model card to a professional standard. CZ: reproducibility is a scientific-integrity practice, and this is where a student practices it.
Mentor role
An ML practitioner, research scientist, or academic reviews the reproduction plan at Checkpoint 1 and critiques the model card at Checkpoint 2. School-supervised.
Rubric calibration
R1: a specific, clearly-scoped published claim. R2: environment and preprocessing documented well enough for a third party. R3: the published number is the baseline. R4: robustness probes are systematic, not cherry-picked. R5: a professional-grade model card. R6: distinguishes "I could not reproduce this" from "this is wrong."
Two ways this goes wrong
(a) The student gets 4% below the published number, blames the authors, and never checks the preprocessing. (b) The student searches subgroups until one looks bad and reports it as a fundamental flaw, with no correction for multiple comparisons.
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.