AI projects · Evaluate · weekend

Spot the hallucination: a fact-check protocol

These projects are not aligned to any state standards or curriculum. They are generally about learning, understanding, using, and evaluating AI.

Ages 11-14

Overview

Language models sometimes invent facts, quotes, or citations that sound real. You will design a short fact-check protocol, run it on chatbot answers about a topic you can verify with free public sources, and score where the model was solid, shaky, or wrong.

Age band

Ages 11–14 (middle school).

Time needed

Weekend (about 3–5 hours).

AI goal

Evaluate — catch confident-sounding errors with a repeatable checklist.

Materials

Free tools only; no paid accounts required.

  • A free browser chat assistant (no paid plan).
  • Two free verifiable sources per question (encyclopedia with citations, government page, textbook-on-loan, or library site). If a URL will not open, mark it UNVERIFIED and replace it.
  • A scoring sheet with columns: question, model answer (short quote), check result (supported / unsupported / contradicted), source used.

Pick a topic with stable facts (state capitals, basic physics definitions, published sports final scores from a completed season, public historical dates). Avoid medical or legal advice topics.

Steps

  1. Write five factual questions with answers you can verify offline or on an official page. Example: "What is the boiling point of water at standard atmospheric pressure in °C?"
  2. Draft the protocol (keep it on one page):
    • Ask the model for an answer and a source.
    • Underline every specific name, number, and date.
    • Check each underlined item against a source.
    • Label each item supported / unsupported / contradicted.
    • Ask one follow-up: "What might be wrong in your answer?"
  3. Run the protocol on all five questions without peeking at sources first for the model's answer (you may peek when checking).
  4. Tally. Count underlined claims; compute the fraction contradicted and the fraction unsupported.
  5. Rewrite one answer in your own words using only supported claims. Put unsupported claims in a "parked" list.
  6. Reflect. When did the model sound most confident? Did confidence match correctness in your five trials? (Answer from your data only.)

Ethics and safety note

Do not use the protocol to spread panic about AI; use it to practice careful reading. Do not ask the model for help harming people or bypassing school rules. Do not submit model text as your own schoolwork.

What done looks like

CriterionNot yetDone
ProtocolAd hoc clickingOne-page checklist used on every question
CoverageOne questionFive questions, claims underlined and labeled
MathAnecdotes onlyFraction contradicted and unsupported reported
RewriteStill contains unchecked claimsFinal paragraph uses supported claims only

Adaptations

Classroom

Pairs fact-check different questions; class merges tallies on the board.

Homeschool

Use a shelf encyclopedia as the second source to practice non-web verification.

Weekend

Three questions instead of five if needed; keep the full protocol.

Research

Compare two free chat tools on the same five questions; report both tallies.

Science fair

Show a before/after: raw model answer vs your supported-only rewrite.

All AI projects

Spot the hallucination: a fact-check protocol · Schoolhouse