AI projects · Evaluate · weekend
Spot the hallucination: a fact-check protocol
These projects are not aligned to any state standards or curriculum. They are generally about learning, understanding, using, and evaluating AI.
Ages 11-14
Overview
Language models sometimes invent facts, quotes, or citations that sound real. You will design a short fact-check protocol, run it on chatbot answers about a topic you can verify with free public sources, and score where the model was solid, shaky, or wrong.
Age band
Ages 11–14 (middle school).
Time needed
Weekend (about 3–5 hours).
AI goal
Evaluate — catch confident-sounding errors with a repeatable checklist.
Materials
Free tools only; no paid accounts required.
- A free browser chat assistant (no paid plan).
- Two free verifiable sources per question (encyclopedia with citations, government page, textbook-on-loan, or library site). If a URL will not open, mark it
UNVERIFIEDand replace it. - A scoring sheet with columns: question, model answer (short quote), check result (supported / unsupported / contradicted), source used.
Pick a topic with stable facts (state capitals, basic physics definitions, published sports final scores from a completed season, public historical dates). Avoid medical or legal advice topics.
Steps
- Write five factual questions with answers you can verify offline or on an official page. Example: "What is the boiling point of water at standard atmospheric pressure in °C?"
- Draft the protocol (keep it on one page):
- Ask the model for an answer and a source.
- Underline every specific name, number, and date.
- Check each underlined item against a source.
- Label each item supported / unsupported / contradicted.
- Ask one follow-up: "What might be wrong in your answer?"
- Run the protocol on all five questions without peeking at sources first for the model's answer (you may peek when checking).
- Tally. Count underlined claims; compute the fraction contradicted and the fraction unsupported.
- Rewrite one answer in your own words using only supported claims. Put unsupported claims in a "parked" list.
- Reflect. When did the model sound most confident? Did confidence match correctness in your five trials? (Answer from your data only.)
Ethics and safety note
Do not use the protocol to spread panic about AI; use it to practice careful reading. Do not ask the model for help harming people or bypassing school rules. Do not submit model text as your own schoolwork.
What done looks like
| Criterion | Not yet | Done |
|---|---|---|
| Protocol | Ad hoc clicking | One-page checklist used on every question |
| Coverage | One question | Five questions, claims underlined and labeled |
| Math | Anecdotes only | Fraction contradicted and unsupported reported |
| Rewrite | Still contains unchecked claims | Final paragraph uses supported claims only |
Adaptations
Classroom
Pairs fact-check different questions; class merges tallies on the board.
Homeschool
Use a shelf encyclopedia as the second source to practice non-web verification.
Weekend
Three questions instead of five if needed; keep the full protocol.
Research
Compare two free chat tools on the same five questions; report both tallies.
Science fair
Show a before/after: raw model answer vs your supported-only rewrite.