Adversarial ML & AI Security · Capstone brief
SEC-08 · Secrets in the context window
On a school-built RAG demo stuffed with fake sensitive documents (teacher provides the corpus: dummy IEPs, dummy passwords, dummy medical notes — all clearly synthetic), how often does the assistant reveal retrieved text it was told not to — and does a retrieval or output filter reduce that rate without wrecking useful answers?
School-approved sandbox only. No production targets. Showcase materials paraphrase class behavior — do not publish payload strings.
The question
On a school-built RAG demo stuffed with fake sensitive documents (teacher provides the corpus: dummy IEPs, dummy passwords, dummy medical notes — all clearly synthetic), how often does the assistant reveal retrieved text it was told not to — and does a retrieval or output filter reduce that rate without wrecking useful answers?
Lab / materials
Local RAG the teacher stands up. No real student records, ever. Fake docs must be obviously fake (names like “SAMPLE STUDENT A”). Benign question set = ordinary “when is the assignment due” classroom queries.
Expected failure modes
Using a production knowledge base. Measuring only leak success, not useful-answer survival. Publishing the dummy “secrets” as if they were real.
Done looks like
Leak rate with/without one filter, useful-answer rate on the benign set, and a one-page recommendation (filter, refuse-to-retrieve, or “do not put this class of doc in the index”). Refusal log.
Five C's
CT: retrieval is the bug, not “the model is evil.” CR: a filter a teacher could toggle. CO: peer writes five new questions you did not anticipate. CM: a data-handling note for the media specialist. CZ: real IEPs do not belong in this lab.
Mentor role
Privacy officer, librarian, or app-sec engineer reviews corpus rules at Checkpoint 1. School-supervised.
Rubric calibration
R1: synthetic corpus only. R2: leak definition written down. R3: unfiltered RAG is the baseline. R4: utility vs leak tradeoff. R5: recommendation is operational. R6: zero real PII.
Two ways this goes wrong
(a) The student indexes a class Google Drive “to make it real.” (b) Perfect leak catch because the bot now answers nothing.
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.