Adversarial ML & AI Security · Capstone brief
SEC-01 · Prompt injection catalog
Against a school-approved sandboxed assistant, can you (1) classify observed failures into a small published taxonomy, (2) measure how often each class succeeds under a fixed trial budget, and (3) show whether one boring defense actually reduces success rate — without pretending the catalog is complete or production-ready?
School-approved sandbox only. No production targets. Showcase materials paraphrase class behavior — do not publish payload strings.
The question
Prompt injection is a family of failures, not a single prompt. Against a school-approved sandboxed assistant, can you (1) classify observed failures into a small published taxonomy, (2) measure how often each class succeeds under a fixed trial budget, and (3) show whether one boring defense actually reduces success rate — without pretending the catalog is complete or production-ready?
Lab / materials
Teacher names one sandbox in writing before Checkpoint 1: local/school-hosted assistant with logging, an education-tier playground the school has approved, or an intentionally vulnerable instructional lab the cyber program already runs.
Taxonomy source (concepts only — not a payload pack):
- OWASP Top 10 for LLM Applications, especially prompt-injection and insecure-output-handling families (https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/)
- The teacher’s class handout or a single approved vendor advisory that names classes. Students do not scrape public “jailbreak lists” into the turn-in.
Defense to measure (pick one, keep it small):
- Delimiters / untrusted-content wrapping if the platform supports it,
- A coarse input policy the teacher can actually deploy (length, role, or tool-use confirmation),
- Or a human review gate on a named sensitive action (the sandbox must expose a fake “action” to gate — send-email, open-ticket, fetch-url — not a real one).
Expected failure modes
Building a witty prompt zoo instead of a taxonomy. Measuring only attack success and ignoring blocked legitimate teacher questions. Changing model, temperature, and defense at once. Publishing class examples detailed enough to be a cookbook. Testing a production helpdesk bot “because it is right there.”
Done looks like
A catalog packet:
- Scope sheet — approved sandbox, taxonomy citation, out-of-scope list.
- Class table — 4–8 named classes from the approved taxonomy, each with a one-line behavioral description (what the system does wrong), not a recipe.
- Trial log — fixed budget (teacher sets a cap; suggest 20–40 labeled trials), success/fail rubric, held-constant settings.
- Defense delta — same trial set, one defense on vs off, plus a benign set of ordinary classroom prompts to show false-block cost.
- Limits memo — what this catalog cannot claim about other models, and a refusal log.
Showcase materials paraphrase class behavior. Verbatim trial strings stay in a teacher-only appendix if the school requires them.
Five C's
CT: treating injection as classes with rates, not anecdotes. CR: choosing a defense small enough to measure. CO: a peer independently labels ten transcripts against the class table; agreement is reported. CM: a table a Security+ teacher could teach from. CZ: who is harmed if students copy class tricks onto a live system.
Mentor role
A cybersecurity teacher, SOC analyst, or application-security engineer reviews the class table at Checkpoint 1 (before trials) and the defense-delta write-up at Checkpoint 2. Reject any plan that names a production target. School-supervised.
Rubric calibration
R1: one sandbox, published taxonomy, written approval. R2: procedure repeatable without a secret prompt stash. R3: undefended vs one defense. R4: false-block rate on benign prompts reported. R5: catalog is teachable in one sitting. R6: refusal log names live systems and payload-publishing.
Two ways this goes wrong
(a) The student turns in twenty spicy transcripts and no rates — a scrapbook, not a catalog. (b) The student “validates” classes against the school’s public chatbot and writes it up as rigor; the brief fails on R6 regardless of the table quality.
Checkpoint suggestions
- Week 1–2: Scope sheet signed; taxonomy frozen at 4–8 classes; trial budget set.
- Week 4–5: Trial log + benign set complete; peer labeling done.
- Week 7–8: Defense delta and limits memo; showcase version stripped of trial strings.
Credit lane fit
Lane A immediately. Lane B only with a division Internship wrapper. No verified credit claim.
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.