Adversarial ML & AI Security · Capstone brief

SEC-02 · Poison in the training set

Using a teacher-provisioned small-classifier lab (public toy dataset + an approved educational notebook or harness), can a small fraction of poisoned labels flip the model on a pre-specified trigger condition from a published paper’s description — and what happens to clean accuracy when it does?

← All briefsIn the bank PDF · use Print

School-approved sandbox only. No production targets. Showcase materials paraphrase class behavior — do not publish payload strings.

The question

Using a teacher-provisioned small-classifier lab (public toy dataset + an approved educational notebook or harness), can a small fraction of poisoned labels flip the model on a pre-specified trigger condition from a published paper’s description — and what happens to clean accuracy when it does?

Lab / materials

Teacher provides the harness. Students do not design a new poisoning method. Suitable public toys (confirm before assign): scikit-learn digits or a small public text set the school already uses. Trigger condition and poison budget are fixed in the lab brief the teacher hands out, drawn from a published educational description — not invented by the student.

Expected failure modes

Treating overall accuracy as the story (the point is the split: trigger vs clean). Poisoning a dataset that leaves the lab. Claiming a production recommendation system is now “pwned.” Running the lab on anything with real student records.

Done looks like

A one-page measurement: poison fraction, clean accuracy, trigger-condition accuracy, and a short statement of whether a simple data-audit (duplicate hunt, label-flip rate, outlier review) would have caught it. Refusal log required.

Five C's

CT: the tradeoff is the finding. CR: proposing an audit a teacher could run Monday. CO: a peer re-runs the published harness and compares numbers. CM: a chart a non-ML cyber teacher can read. CZ: who is harmed if training data is untrusted.

Mentor role

Data scientist or ML engineer reviews the measurement plan at Checkpoint 1 and the audit proposal at Checkpoint 2. School-supervised.

Rubric calibration

R1: teacher harness only; no novel method. R2: numbers regenerable from the lab. R3: clean vs trigger split present. R4: audit catch-rate discussed honestly (including “would have missed it”). R5: one sitting. R6: no production data, no published poison recipe.

Two ways this goes wrong

(a) The student reports 94% accuracy and never shows the trigger split. (b) The student generalizes from a 200-row toy to “hospitals should not use ML.”


Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.