Adversarial ML & AI Security · Capstone brief
SEC-03 · Model extraction on a budget
Against a school-approved black-box demo (local model wrapped as an API, or a lab endpoint the teacher names), how much of the observable behavior can a student match with a fixed query budget — and what does that imply for how a real API should be priced, rate-limited, or logged?
School-approved sandbox only. No production targets. Showcase materials paraphrase class behavior — do not publish payload strings.
The question
Against a school-approved black-box demo (local model wrapped as an API, or a lab endpoint the teacher names), how much of the observable behavior can a student match with a fixed query budget — and what does that imply for how a real API should be priced, rate-limited, or logged?
Lab / materials
Teacher sets the query cap (suggest a hard number the student cannot exceed). The student trains or prompts a surrogate only on allowed queries to the lab endpoint. Public documentation for the pricing discussion: one vendor’s published API pricing page the teacher names at assign time. No scraping of unpaid production APIs.
Expected failure modes
Burning an unpaid commercial API. Calling extraction “stealing the weights” when the student matched a few input-output pairs. Ignoring that the surrogate fails off the query distribution. Treating the write-up as a how-to for ripping a competitor’s model.
Done looks like
Budget ledger, agreement metric on a held-out lab set the teacher provides, a plot or table of fidelity vs queries, and a one-page implication memo for rate limits / logging / pricing. Refusal log: no production APIs, no weight-dump claims.
Five C's
CT: fidelity is not theft of weights. CR: choosing a metric that matches the lab’s task. CO: peer checks the ledger against logs. CM: a pricing/rate-limit memo a product owner could read. CZ: who pays when an API is scraped.
Mentor role
ML engineer or API product person reviews budget design at Checkpoint 1 and the memo at Checkpoint 2. School-supervised.
Rubric calibration
R1: named lab endpoint, hard budget. R2: ledger matches logs. R3: a naive baseline (e.g. majority class / copy-the-prompt) exists. R4: off-distribution failure named. R5: memo is about controls, not a recipe. R6: no commercial API abuse.
Two ways this goes wrong
(a) The student queries a public chatbot thousands of times and calls it research. (b) The student reports “I extracted the model” because two answers matched.
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.