PROV-01 · Detection has a shelf life

Take one publicly available synthetic-media detector and measure it against a ground-truth set you built. How often does it miss synthetic media, how often does it accuse honest media — and when does your own finding expire? Stamp the result with a date and say when a newsroom would have to re-run it.

The interesting number here is not accuracy. It is the false positive rate on real media, because in a newsroom that error is the one that kills a true story and libels an honest source.

Teacher approves the detector, the ground-truth set, and its sources in writing before Checkpoint 1.

• A ground-truth set of at least thirty items, split between known-real and known-synthetic, where known means documented — not assumed, not "it looks real." Every item carries a provenance note saying how its status was established.
• Where the synthetic half comes from, in order of preference: a published research corpus the teacher approves and can actually access; publicly released example sets from a generator's own documentation; or material the students generate themselves — of objects, places, or invented scenes only. The consent and likeness rule governs absolutely: no synthetic person, no classmate, no public figure, not even to test a detector. If the only way to test a face detector is to synthesize a face, the honest deliverable is a written account of why the test could not be run within the rule, and what a professional lab would need to run it properly. That is a finding.
• The real half from your own program's archive where possible — footage and photographs whose capture you can document. This matters: detectors that behave well on clean stock imagery often flag ordinary phone video shot in bad light.
• The detector's own documentation — what it claims to detect, what it was trained on, and its stated version and date. If the tool will not tell you, that is itself a result and belongs in the report.

Do not run any real person's media through a detector and publish or circulate the verdict. Testing is conducted on your set, and results describe the detector, never a person.

Reporting one accuracy figure and stopping — a detector that is 90% accurate on a balanced set can still be useless if the 10% is concentrated in real media. Building a synthetic half from one generator, which measures the detector against that generator and nothing else. Assuming the real half is real because it looks real. Testing on pristine images when the newsroom's actual inputs are compressed, cropped, screenshotted, and re-uploaded. Treating a confidence percentage as a probability of fakery — it is a model output, not a likelihood. Publishing a finding with no date, which is how a measurement becomes folklore.

A detector report card with five pieces:

1. Ground-truth set description — counts, sources, and how each item's status was established. Includes what you could not obtain and why.
2. Results table — false negatives and false positives reported separately as counts, not merged into accuracy, and broken out by condition: original versus re-compressed, clean versus low-light, and by generator or era where the set supports it.
3. The false-positive harm paragraph — pick your worst false positive and write what would have happened if a newsroom had acted on it: whose work was called fake, what the correction would cost, who would not be believed next time.
4. Use protocol — how, if at all, this tool belongs in a verification workflow. Written as an input to human judgment with named limits, never as a verdict. If the answer is "it does not belong yet," say that.
5. Expiry stamp and refusal log — the date of the test, the tool version, the generators represented, an explicit re-test by date with the reason for that interval, and the claims you declined to make.

Ground-truth material stays on school-managed storage. The showcase version reports results and shows no synthetic media of any person.

CT: separating a model's confidence from a claim about the world. CR: designing a set that could actually defeat the tool, including the awkward real media. CO: a peer scores ten items blind from the raw files and the two sets of calls are compared before either is final. CM: a report card an editor could act on in one sitting. CZ: the honest photographer whose work gets called synthetic, and what that costs them.

A photo editor, media forensics researcher, standards editor, or working journalist reviews the ground-truth set at Checkpoint 1 — before testing, since a set built badly cannot be rescued by analysis — and the use protocol at Checkpoint 2. Standing instruction: reject any set containing a synthesized person, and reject any conclusion phrased as "this detector proves." School-supervised, both times.

R1: one detector, one set, one claim. R2: another student could rebuild the set from the description and re-run it. R3: comparator is human judgment on the same items without the tool. R4: false positives and false negatives counted separately and by condition. R5: the report card is decision-ready for an editor. R6: expiry stamp present; refusal log names declined claims.

(a) A single accuracy percentage with no separation of error types, no conditions, and no date — which tells an editor nothing and will be quoted for years. (b) The synthetic half all comes from one generator, so the report measures a matchup rather than a detector, and says so nowhere.

• Week 1–2: Detector and set sources approved; consent and likeness rule signed; ground-truth set assembled with provenance notes; the "cannot obtain" list started.
• Week 4–5: Testing complete with results logged per item; peer blind-scoring of ten items; conditions comparison run on re-compressed copies.
• Week 7–8: False-positive harm paragraph written; use protocol drafted and mentor-reviewed; expiry stamp and refusal log finished.

Lane A immediately (journalism or broadcast capstone, student publication project, or AP Research). Pairs with PROV-11 for a student who wants to argue the process-controls case. No verified credit claim.