AI projects · Evaluate · weekend

Can you trust an AI detector?

These projects are not aligned to any state standards or curriculum. They are generally about learning, understanding, using, and evaluating AI.

Ages 11-14

Overview

AI-writing detectors claim they can tell whether a passage was written by a person or by a language model. In this project you build a small, honest test set and measure how often a free detector is right, wrong, or unsure — then you write what that result does and does not prove.

Age band

Ages 11–14 (middle school). An adult helps create the AI samples and reviews any tool accounts.

Time needed

Weekend (about 3–6 focused hours).

AI goal

Evaluate — judge a tool's claims with a controlled mini-experiment, not vibes.

Materials

Free tools only; no paid accounts required.

  • A free AI-writing detector that works in a browser without a paid plan (for example GPTZero's free check, or another free detector your adult approves). If a tool asks for payment to see results, stop and switch tools.
  • A free chat assistant that works without a paid subscription (browser free tier), used only to generate AI samples.
  • A spreadsheet (Google Sheets, LibreOffice Calc, or paper table) with columns: sample ID, source (human / AI), topic, detector label, detector confidence (if shown), your notes.
  • Six short human-written paragraphs you or classmates already wrote for school (no private journals; ask before using someone else's work).
  • Six AI-written paragraphs on the same topics, generated with a free chat tool, then lightly edited so length is similar.

Do not upload other people's private writing. Prefer your own homework drafts or public-domain text.

Steps

  1. Write the question. Example: "On this six-human / six-AI sample set, how often does Detector X label human text as AI, and AI text as human?"
  2. Build the sample set. Pair each human paragraph with an AI paragraph on the same topic and roughly the same word count (aim for 80–150 words each). Number them S1–S12. Shuffle the order so you do not always test humans first.
  3. Blind yourself. Put the true labels in a sealed note or a hidden sheet column. On the working sheet, show only the text and an ID.
  4. Run the detector once per sample. Record the label and any score exactly as shown. Do not re-run until you have all twelve results (re-running can change answers and muddy the count).
  5. Score the experiment. Compute:
    • True positives (AI text labeled AI)
    • False positives (human text labeled AI)
    • True negatives (human text labeled human)
    • False negatives (AI text labeled human)
    • Accuracy = (true positives + true negatives) / 12
  6. Stress test one edge case. Take one human paragraph and paraphrase it with the free chat tool "in a more formal tone," then re-check. Note whether the label flips. That is one data point, not a new full study.
  7. Write the claim boundary. In one paragraph, state what your accuracy number measures (this detector, this sample set, this day) and what it does not measure (all detectors, all topics, all grade levels, "cheating" in general).

Ethics and safety note

Detectors can falsely accuse people. Never use your results to accuse a classmate of cheating. Do not paste personally identifiable homework that includes names, addresses, or health details into a third-party site. Prefer text you wrote yourself.

What done looks like

CriterionNot yetDone
Sample setFewer than 12 labeled samples, or topics unmatched6 human + 6 AI, topics paired, IDs listed
Blind runLabels visible while testingLabels hidden until all scores recorded
MathOnly a gut feelingConfusion counts + accuracy fraction shown
Claim boundary"Detectors don't work" / "Detectors work" as a universalExplicit limits: tool, sample, date

Adaptations

Classroom

Pairs share one detector account under teacher rules. Compare class accuracy ranges without naming students whose writing was used.

Homeschool

Use family writing samples with permission. Discuss why a false positive would be unfair before looking at scores.

Weekend

Skip the paraphrase stress test if short on time; keep the 12-sample core.

Research

Add a second free detector and compare agreement (both wrong vs one wrong). Still do not generalize beyond your sample.

Science fair

Poster: research question, method schematic, confusion table, accuracy, and a large "What this does not prove" box.

All AI projects

Can you trust an AI detector? · Schoolhouse