Adversarial ML & AI Security · Capstone brief
SEC-02 · Poison in the training set
Using a teacher-provisioned small-classifier lab (public toy dataset + an approved educational notebook or harness), can a small fraction of poisoned labels flip the model on a pre-specified trigger condition from a published paper’s description — and what happens to clean accuracy when it does?
School-approved sandbox only. No production targets. Showcase materials paraphrase class behavior — do not publish payload strings.
The question
Using a teacher-provisioned small-classifier lab (public toy dataset + an approved educational notebook or harness), can a small fraction of poisoned labels flip the model on a pre-specified trigger condition from a published paper’s description — and what happens to clean accuracy when it does?
Lab / materials
Teacher provides the harness. Students do not design a new poisoning method. Suitable public toys (confirm before assign): scikit-learn digits or a small public text set the school already uses. Trigger condition and poison budget are fixed in the lab brief the teacher hands out, drawn from a published educational description — not invented by the student.
Expected failure modes
Treating overall accuracy as the story (the point is the split: trigger vs clean). Poisoning a dataset that leaves the lab. Claiming a production recommendation system is now “pwned.” Running the lab on anything with real student records.
Done looks like
A one-page measurement: poison fraction, clean accuracy, trigger-condition accuracy, and a short statement of whether a simple data-audit (duplicate hunt, label-flip rate, outlier review) would have caught it. Refusal log required.
Five C's
CT: the tradeoff is the finding. CR: proposing an audit a teacher could run Monday. CO: a peer re-runs the published harness and compares numbers. CM: a chart a non-ML cyber teacher can read. CZ: who is harmed if training data is untrusted.
Mentor role
Data scientist or ML engineer reviews the measurement plan at Checkpoint 1 and the audit proposal at Checkpoint 2. School-supervised.
Rubric calibration
R1: teacher harness only; no novel method. R2: numbers regenerable from the lab. R3: clean vs trigger split present. R4: audit catch-rate discussed honestly (including “would have missed it”). R5: one sitting. R6: no production data, no published poison recipe.
Two ways this goes wrong
(a) The student reports 94% accuracy and never shows the trigger split. (b) The student generalizes from a 200-row toy to “hospitals should not use ML.”
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.