Adversarial ML & AI Security · Capstone brief
SEC-05 · Detection that is not theater
For one attack class already named in SEC-01’s taxonomy (or a teacher-provided class list), can you design a detector whose false-positive cost on ordinary teacher prompts is explicit — and decide whether you would actually turn it on?
School-approved sandbox only. No production targets. Showcase materials paraphrase class behavior — do not publish payload strings.
The question
For one attack class already named in SEC-01’s taxonomy (or a teacher-provided class list), can you design a detector whose false-positive cost on ordinary teacher prompts is explicit — and decide whether you would actually turn it on?
Lab / materials
Same approved sandbox as SEC-01, or a transcript corpus the teacher generates from benign classroom use plus labeled lab failures. No production logs. No scanning student email.
Expected failure modes
A detector with 100% recall because it flags everything long, foreign, or slightly weird. Reporting precision without a benign set. Claiming “we blocked jailbreaks” on ten attack transcripts and zero teacher questions.
Done looks like
Detector description (rules or simple model — keep it boring), confusion-style counts on attack-lab vs benign-teacher sets, a recommended threshold with the FP budget stated in “prompts per class period,” and a go/no-go. Refusal log.
Five C's
CT: FP budget is the assignment. CR: a detector a teacher could run. CO: peer supplies additional benign prompts you did not write. CM: a one-page ops note. CZ: over-blocking silences legitimate student questions.
Mentor role
Detection engineer or cyber teacher reviews the benign set at Checkpoint 1 and the go/no-go at Checkpoint 2. School-supervised.
Rubric calibration
R1: one class. R2: both sets documented. R3: “flag everything” is the baseline to beat. R4: FP cost in classroom units. R5: ops note is usable. R6: no production log vacuuming.
Two ways this goes wrong
(a) Perfect attack catch, 40% of real homework flagged, called a success. (b) Detector is “I know it when I see it” with no counts.
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.