SEC-04 · Jailbreak, then report

Public reporting has documented classes of jailbreak and safety-bypass behavior against language-model assistants (role-play framing, instruction hierarchy conflicts, obfuscation, multi-turn goal drift, and related patterns). In a school-approved sandbox, can you (1) reproduce one published class under controlled conditions, (2) show that a simple defense reduces success rate relative to an undefended baseline, and (3) write the incident note a security team would file — including what you refused to attempt?

Pick one approved target the teacher names in writing before Checkpoint 1, for example:

• A local or school-hosted assistant with a documented system prompt and logging enabled, or
• A vendor “playground” / education tier the school has approved for security labs, or
• An intentionally vulnerable instructional LLM lab the cyber program already uses.

Do not use the district production helpdesk bot, gradebook tools, or any system that processes real student PII.

Reference material (concepts and taxonomies — not a payload pack):

• OWASP Top 10 for LLM Applications (prompt injection / insecure output handling families) (https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/)
• NIST AI Risk Management Framework overview (map your incident to a risk category in plain language) (https://www.nist.gov/itl/ai-risk-management-framework)
• One already published write-up or vendor advisory the teacher approves that describes the class you will test (paper, vendor blog, or class handout). Students cite it; they do not scrape random “jailbreak lists” into the turn-in.

Optional defense baseline to compare against (pick one, keep it boring):

• Input filter / deny-list for known framing markers the class uses,
• Stronger system-prompt separation (“untrusted user content” delimiters) if the platform allows,
• Or a human-in-the-loop rule: tool use / sensitive actions require confirmation.

Treating a single funny transcript as proof the model is “fully jailbroken.” Changing three variables at once (model, prompt, temperature) so nothing is attributable. Copying a full jailbreak string from the internet into a shared drive or public GitHub. Declaring a defense “works” on five prompts. Testing against a production system “just to see.” Measuring only attack success and ignoring that the defense blocks legitimate teacher questions.

A red-team packet with four pieces:

1. Scope sheet — approved target, attack class name, citation for the published class, explicit out-of-scope list.
2. Method — how many trials, what counted as success/failure, what was held constant, what defense was applied in the treatment condition.
3. Results — success rate with and without the defense (or qualitative rubric if the sandbox forbids automation), plus at least one benign prompt set showing whether the defense is theater.
4. Incident note (one to two pages) — what happened, impact if this were production, detection signals, recommended mitigation, and a refusal log of actions not taken.

No requirement to include verbatim attack strings in the student portfolio that leaves the school network. Teachers may require a private appendix; public showcase materials use paraphrased class descriptions.

CT: separating “model said something edgy once” from a reproducible class failure. CR: designing a minimal defense worth measuring, not a fifteen-layer fantasy stack. CO: a peer re-scores five transcripts blind against your success rubric. CM: an incident note a non-ML security teacher can file. CZ: naming who gets hurt if jailbreak techniques leave the lab (students, staff, vulnerable users of the real system).

A cybersecurity practitioner, SOC analyst, or secure-ML engineer reviews the scope sheet at Checkpoint 1 (before any trials) and the incident note at Checkpoint 2. Standing instruction: reject any plan that names a production target. School-supervised, both times.

R1: one class, one target, written approval. R2: another student could repeat the procedure without a secret prompt stash. R3: undefended vs defended comparison exists. R4: false-positive / legitimate-use impact addressed. R5: incident note is actionable in one sitting. R6: refusal log has real items (not “I refused to hack the Pentagon”).

(a) The student pastes a viral jailbreak, gets a spicy answer, screenshots it, and writes “models are unsafe” — no baseline, no defense, no class name. (b) The student tests the school’s public-facing chatbot without approval and treats the write-up as a capstone; that is a conduct issue, and the brief is failed on R6 regardless of technical cleverness.

• Week 1–2: Scope sheet signed. Attack class chosen from teacher-approved references only.
• Week 4–5: Trial log complete; defense condition run; peer blind-score done.
• Week 7–8: Incident note + refusal log; showcase version stripped of any sensitive strings.

Lane A immediately (cyber senior project / Security+ aligned culminating task). Lane B only if the division wraps it as Internship with a real host and training plan — the brief alone is not HQWBL. No verified credit claim.