Representation: Numbers, Vectors, Embeddings

Big idea: A computer cannot hold a meaning. It holds numbers. So before any model can translate a sentence, caption a photo or finish your code, somebody has to answer a hard design question: what numbers should stand for this thing? That choice is called a representation, and almost everything modern AI can — and cannot — do traces back to it.

Everything becomes numbers first

Three examples, all of which you already use every day:

• Text. A tokenizer chops a string into pieces called tokens — roughly words, but common words stay whole and rare words split up (unbelievable → un + believ + able). Every token in the model's fixed vocabulary has an integer ID. The sentence you type leaves your keyboard as prose and arrives at the model as a list of integers like [8, 214, 3, 1097].
• Images. A photo is a grid of pixels. Each pixel is three integers from 0 to 255 — how much red, green and blue. A 224 × 224 colour image is therefore an array of 224 × 224 × 3 = 150,528 numbers. The cat in the picture is nowhere in that array. Only brightness values are.
• Audio. A microphone measures air pressure thousands of times a second — CD-quality audio takes 44,100 samples per second. One second of sound is 44,100 numbers on a line.

Nothing here is intelligent yet. This is just a rule for turning stuff into numbers. But which rule you pick decides what the model can learn.

One-hot vectors, and why they dead-end

Here is the obvious first idea. Give every word in your vocabulary its own slot. If the vocabulary is [cat, dog, kitten, car, truck], then:

cat    = [1, 0, 0, 0, 0]
dog    = [0, 1, 0, 0, 0]
kitten = [0, 0, 1, 0, 0]

That is a one-hot vector — one 1, the rest 0s. It is unambiguous, it is easy to build, and it is a dead end for two reasons.

It is enormous. A real vocabulary is 50,000+ tokens, so every single word is a 50,000-dimensional vector that is 99.998% zeros.

It knows nothing. Compute the dot product of any two different one-hot vectors and you get 0 — every time, for every pair. cat is exactly as unrelated to kitten as it is to truck. The representation contains no notion of similarity at all, so a model built on it has to learn every word from scratch, with nothing transferring between them.

Dense embeddings: the actual leap

The fix is to stop using one slot per word and start using a few hundred numbers, all of them meaningful. That is an embedding: a dense vector, typically 300 to a few thousand dimensions, where no single dimension means anything on its own but the combination places the word somewhere specific.

Somewhere specific in what? In a space. A vector of two numbers is a point on a plane; a vector of three is a point in a room; a vector of 768 is a point in a 768-dimensional space you cannot picture and do not need to. The geometry still works. Words with similar meanings end up near each other, and that is the whole payoff — similarity becomes something you can measure.

Measuring closeness

Take a deliberately tiny 2-D space and check the arithmetic by hand:

Word	Vector
cat	(3, 4)
dog	(4, 3)
kitten	(6, 8)
car	(−4, 3)
truck	(−3, 4)

Cosine similarity measures the angle between two vectors, ignoring their length:

                a · b            (a₁b₁ + a₂b₂ + … + aₙbₙ)
cos(a, b) = ───────────  =  ────────────────────────────────
              |a| × |b|      √(Σaᵢ²)  ×  √(Σbᵢ²)

For cat and dog: the dot product is (3)(4) + (4)(3) = 24, and both lengths are 5, so cos = 24 / 25 = 0.96. Close. For cat and car: (3)(−4) + (4)(3) = 0, so cos = 0.00 — perpendicular, unrelated. For cat and kitten: (3)(6) + (4)(8) = 50, lengths 5 and 10, so cos = 50 / 50 = 1.00 — the same direction, even though kitten's vector is twice as long and straight-line distance between them is 5. That contrast is the point: direction carries the meaning; length usually carries something else, like how often the word appears. Cosine runs from +1 (same direction) through 0 (unrelated) to −1 (opposite).

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
    return dot / mag

That five-line function is, genuinely, how a search engine decides two documents are about the same thing and how a recommender decides you might like something.

The famous trick — and the honest footnote

If meanings are positions, then differences between positions might be relationships. Build a space where the first coordinate tracks royalty and the second tracks a gender association:

man = (2, 1) woman = (2, 3) king = (6, 1) queen = (6, 3)

Then king − man + woman = (6, 1) − (2, 1) + (2, 3) = (6, 3) = queen. Analogy by arithmetic. It is a real effect and it genuinely appears in embeddings trained on ordinary text.

Now the footnote nobody puts on the poster. The result is usually not exactly queen — it is a point near queen, and the system returns the nearest word to it. Standard practice excludes the three input words from the answer, which quietly removes the most common wrong answers; without that rule, the nearest word to the result is often king itself. And the trick works far better for some relations (capital cities, verb tenses, plurals) than for others, where it produces confident nonsense. The demo is evidence that structure exists in the space. It is not evidence that the model understands kings.

Where the numbers come from

Nobody typed these vectors in. They are learned, from a single stubborn observation known as the distributional hypothesis: words that show up in similar contexts tend to mean similar things. Feed a system enormous amounts of ordinary text, repeatedly nudge each word's vector toward the vectors of the words it appears near and away from the words it doesn't, and after billions of adjustments the geometry falls out on its own. cat and kitten end up close because they get used in the same sentences — never because anyone told the machine what a cat is.

Which means the space inherits us

If the vectors come from what people actually wrote, then whatever patterns are in that writing end up in the geometry. When a corpus overwhelmingly places nurse near she and surgeon near he, the embedding encodes that as a direction in space — and any system built on top of it will inherit it, quietly, at scale. This is not a bug in the arithmetic. The arithmetic worked perfectly; it measured the text it was given.

It also means the fix is not simple. You can mathematically subtract a "gender direction" from the vectors, and researchers have shown that afterwards the biased words are still clustered together — the association has been made harder to see rather than removed. Fixing the numbers is not the same as fixing what they represent, which is why every later unit in this course keeps a human in the loop.

Essential question: If a machine only ever sees numbers, where does meaning actually live — and who put it there?

Teacher Guide — Representation: Numbers, Vectors, Embeddings

Sessions: 2 × 55 min (or 3 × 45; see pacing) · Format: the core lesson is fully unplugged — every required beat runs on printed mats and hand arithmetic with 2-D vectors. One optional extension needs devices (a spreadsheet or a Python interpreter, one per pair); it is marked clearly and nothing in the assessment depends on it. No student data is entered into any AI tool at any point in this unit.

At a glance

• Big idea: A computer stores numbers, never meanings. A representation is the rule that turns a thing into numbers, and the quality of that rule sets the ceiling on everything a model can learn. One-hot vectors are unambiguous and useless — every pair of distinct words is equally dissimilar. Dense embeddings fix that by making meaning a position, so similarity becomes a measurable angle.
• Essential question: If a machine only ever sees numbers, where does meaning actually live — and who put it there?
• Why this is Unit 1. Every remaining unit in this band assumes it. Training (U2) is adjusting these vectors. Attention (U3) is a weighted sum of these vectors. LLMs (U4) predict the next token ID. Retrieval (U8) is literally cosine similarity over embeddings. Bias at scale (U9) is this unit's last section, industrialised. If students leave with one thing, make it: the model never sees the word.
• You do not need a linear algebra background to teach this. Everything students compute is 2-D: multiply, add, one square root. The full worked arithmetic is in the answer key of this unit's Printable Materials.

Learning targets (student language)

1. I can take a piece of text, an image or a sound and describe exactly how it becomes numbers.
2. I can build a one-hot vector and explain, using the dot product, why it can't represent similarity.
3. I can compute cosine similarity between two vectors by hand and say what the number means.
4. I can explain why direction matters more than length in an embedding space.
5. I can do analogy arithmetic on vectors and state one honest limitation of the result.
6. I can explain that embeddings are learned from co-occurrence, and say where bias in them comes from.

Standards

This unit sits in Computer Science Foundations (CSF) — the course in Virginia's 2024 CS SOL that carries the explicit machine-learning and neural-network content for grades 9-12. Virginia does not define grade-level 9-12 CS standards; it defines courses. No individual CSF codes are claimed for this unit. Course placement is the alignment claim; codes stay unassigned rather than invented. Units 2, 3 and 4 of this band carry the verified CSF codes that those units earn.

VDOE AI Guidance: Empower Student Success (students reason about how AI systems actually represent information, rather than treating them as magic), Augment Not Replace (the closing bias discussion establishes that measuring a pattern is not the same as endorsing it, and that a person stays accountable for what a representation is used for).

Materials

• Mat A (Representation pipeline), Mat B (One-hot grid), Mat C (2-D meaning space), Mat D (Cosine worksheet), Card set E (Analogy cards) and Card set F (Bias audit) — all printable from this unit's Printable Materials. One set per pair.
• Rulers and a calculator that does square roots (a phone calculator is fine; students do not need the internet for it).
• Graph paper or a projected coordinate grid for the plotting beat.
• Optional, devices needed: one spreadsheet or Python environment per pair for the extension in step 6.

Pacing

Plan	Session 1	Session 2	Notes
Standard (2 × 55)	Steps 1-3 (representation, one-hot, the dead end)	Steps 4-7 (embeddings, cosine, analogy, bias)	The default.
Three shorter periods (3 × 45)	Steps 1-2	Steps 3-5	Steps 6-7 in period 3, with more time on bias.
Honours / CSP-bound	As standard	As standard	Add the extension in step 6 and the second-order bias question in Differentiation.
Compressed (1 × 55)	Steps 1, 2, 4, 5 only	—	Drop analogy arithmetic and the bias audit; assign the take-home to carry them.

If you only have 20 minutes: run the Hook, the one-hot dot product (step 2), and one row of the cosine table on Mat D — cat/dog = 0.96 versus cat/car = 0.00. That single contrast is the load-bearing idea of the unit. Close with the essential question and nothing else.

---

Lesson plan

1. Hook — "send me a smell" (5 min)

Put this on the board: "You have a wire. It carries numbers. Nothing else. Send me the word cat." Take suggestions and write them all up. Students will propose ASCII codes, dictionary positions, a spelling-based scheme, a picture.

Then ask the question that reframes all of them: "Fine — now send me kitten so that I can tell it's related to cat without knowing English." Let the silence sit. That is the entire unit's problem in one sentence.

Why this works: it converts an abstract topic into a design constraint. Students immediately discover that encoding is easy and encoding similarity is not.

2. I do — the pipeline, three ways (10 min) · Mat A

Model all three encodings out loud, quickly, on Mat A. Do not lecture; narrate.

• Text: write I love unbelievable cats on the board, chop it into tokens (I love un believ able cats), and assign each an arbitrary ID. Emphasise that the IDs are arbitrary labels, not quantities — token 4092 is not twice token 2046. Show a rare word splitting and a common word staying whole, and say why: a fixed vocabulary has to cover infinite language, so it stores pieces.
• Image: hold up any photo. 224 × 224 × 3 = 150,528 numbers, each 0-255. Write the multiplication out. Ask: "Which of those numbers is the cat?" (None. That is the answer.)
• Audio: draw a waveform and mark sample points. 44,100 numbers per second.

Land the move: "Three completely different things, one destination — a list of numbers. From here on, the model is doing arithmetic, not reading."

3. We do — one-hot, and killing it (12 min) · Mat B

Build one-hot vectors as a class for the five-word vocabulary on Mat B (cat, dog, kitten, car, truck). Every student can do this; it is a confidence beat.

Then run the kill shot. Have pairs compute the dot product of cat with kitten, then cat with truck. Collect answers. Both are 0. Push: "Compute any two different words you like. Anyone get something other than zero?" Nobody will.

Now name the consequence precisely: in a one-hot representation, every distinct word is exactly as similar to every other word — which is to say, not at all. Add the second problem: 50,000 tokens means 50,000 dimensions, 99.998% of them zero. Ask what a model would have to do to learn that "cat" and "kitten" behave alike (answer: learn each one separately, from scratch, with nothing shared).

This beat is the hinge of the lesson. Do not shorten it. Students who feel one-hot fail are ready for embeddings; students who skip it experience embeddings as an arbitrary complication.

4. We do — meaning as a position (10 min) · Mat C

Introduce the fix as a design decision, not a formula: "Stop giving each word its own slot. Give every word a few hundred numbers, and let similar words land near each other."

Plot the five toy 2-D vectors from Mat C on graph paper together: cat (3, 4) · dog (4, 3) · kitten (6, 8) · car (−4, 3) · truck (−3, 4).

Ask what they notice before you introduce cosine. Students reliably see the two clusters and the fact that kitten points the same way as cat but further out. Bank both observations — you are about to formalise exactly them.

Say the honest caveat now, so nobody carries a misconception forward: "Real embeddings have hundreds of dimensions and no axis means anything on its own. Two dimensions is a teaching model, not a picture of the real thing."

5. You do — cosine similarity by hand (12-15 min) · Mat D

Give the formula, then get out of the way:

cos(a, b) = (a · b) / (|a| × |b|)

Pairs complete the Mat D table. All arithmetic is small integers and clean square roots; full answers are in the materials key. The four rows that matter:

Pair	Cosine	What it tells you
cat · dog	0.96	Very similar meaning.
cat · kitten	1.00	Identical direction — despite being 5 units apart in straight-line distance.
cat · car	0.00	Perpendicular. Unrelated.
dog · car	−0.28	Pointing apart.

Then ask the question that earns the concept: "cat and kitten are 5 units apart but score a perfect 1.00. What did cosine ignore, and why is ignoring it a good idea?" Land it: cosine measures direction, not magnitude. In real embeddings the magnitude tends to track how common or how frequent a token is — not what it means — so throwing it away is a feature.

6. You do — analogy arithmetic, with the footnote (10 min) · Card set E

Pairs lay out man (2, 1), woman (2, 3), king (6, 1), queen (6, 3) and compute king − man + woman. It lands exactly on queen. Let them enjoy it — it is a genuinely delightful result.

Then immediately do the intellectual honesty work, because this is the single most over-claimed demo in popular AI writing. Ask: "Why did that work perfectly here?" (Because the space was built so it would; the axes were chosen.) Then give them the three real limitations:

1. In a real model the result is a point near queen, not queen — the system returns the nearest word to that point.
2. The standard method excludes the three input words from candidate answers. Without that rule, the nearest word to the result is very often king itself.
3. It works well for some relation types (capitals, plurals, verb tenses) and produces confident nonsense for others.

Close the beat with the sentence worth memorising: "This shows there is structure in the space. It does not show the model understands kings."

Optional extension (devices required): pairs implement cosine(a, b) in Python or a spreadsheet and re-run the table, then try 4-D vectors where the hand arithmetic gets tedious. Nothing in the assessment requires this.

7. Closure — the space inherits us (8 min) · Card set F

Ask the question directly: "Nobody typed these vectors in. Where did they come from?" Give them the distributional hypothesis in plain words — words used in similar contexts get similar vectors — and make it concrete: cat and kitten are close because they appear in the same sentences, never because anyone defined a cat.

Then run the Bias audit cards. Each pair takes one card, predicts which words the corpus would place near each other, and answers: who wrote the text this was learned from, and what would that put in the geometry?

Close on the two-part landing:

• The arithmetic was not wrong. It measured the text it was given, correctly.
• And you cannot fix it with more arithmetic alone. Researchers have shown that subtracting a "gender direction" leaves the biased words still clustered together — it makes the association harder to detect rather than removing it. Which is why a person stays accountable for what a representation gets used for.

---

Common misconceptions

• "Each dimension of an embedding means something — like dimension 7 is 'royalty'." → No. In a trained embedding, individual dimensions are generally not interpretable; meaning lives in the combination and in directions through the space. Mat C's readable axes are a scaffold, and you should say so out loud when you introduce it.
• "Token IDs are meaningful numbers." → They are arbitrary labels. Token 900 is not "more" than token 450, and the model does not do arithmetic on IDs — it uses the ID to look up a vector. This is exactly the row-lookup a one-hot vector times a matrix performs, which is worth saying to students who ask why one-hot is taught at all.
• "Cosine similarity measures distance." → It measures angle. Two vectors can be far apart in Euclidean distance and have cosine 1.00 — the cat/kitten row exists to prove it.
• "king − man + woman = queen proves the model understands gender." → It proves the space has linear structure that partly aligns with some relations. See step 6.
• "An embedding is the model." → It is one layer, usually the first. Units 2-4 build everything on top of it.
• "Bias got in because someone was careless with the algorithm." → The algorithm did its job. Bias arrived with the text, and text is written by people. This distinction matters enormously in Unit 9; establish it now.
• "We can just remove the bias mathematically." → See the closure. Reducing a measurement of a pattern is not the same as removing the pattern.

Background (teacher notes)

The reason representation deserves the first unit of the band is that it is the concept that makes every later one inevitable rather than arbitrary. Attention looks like an unmotivated formula until students already believe that a word is a point in space and that closeness is a dot product — at which point attention reads as "how much should each other token's vector contribute to mine?", which is a sentence they already understand.

The one-hot beat is worth defending against the temptation to cut it. One-hot is not a historical curiosity: it is the honest baseline that makes the embedding a solution to something. It also quietly sets up a fact students meet again in Unit 2 — multiplying a one-hot vector by a weight matrix selects a single row, which is precisely what an embedding lookup table is. You do not have to teach that here, but if a strong student notices it, they have found something real.

On the analogy demo: be aware that it is frequently reported without its caveats, so students will have met the confident version online. Teaching the caveats is not deflating the result — the result is real and remarkable. It is teaching them the habit of asking what was excluded from the answer set, which is a transferable form of scepticism about AI claims generally, and which pays off again in Unit 4 when they meet benchmark scores.

On bias: keep it technical, because the technical version is both more accurate and less inflammatory. The claim is not that a model "is sexist". The claim is that a co-occurrence-based representation encodes the statistical regularities of its corpus, including the ones we would not choose to reproduce, and that downstream systems built on it inherit them by default rather than by decision. That framing survives contact with a sceptical student, a sceptical parent and a sceptical administrator.

Equity, privacy & safety note

• No student data goes into any AI system in this unit. The optional extension is local arithmetic in a spreadsheet or a Python interpreter — no accounts, no prompts, no uploads. If you choose to demonstrate a live embedding tool yourself, use neutral words you have chosen in advance; do not type student names, and do not let the class supply the input.
• The bias discussion is technical, not personal. The Bias audit cards are deliberately written about occupations and corpora, not about anybody in the room. If a student raises a personal experience, acknowledge it briefly and honestly, then return to the mechanism. Do not let the class become the example.
• Do not let the analogy demo run on student-supplied identity terms. "king − man + woman" is on the card. Open-ended "type any two identity words and see what it says" is not a lesson, it is an exposure risk, and it teaches nothing the structured version does not.
• Mathematics is not a prerequisite for participating. Every required computation is two multiplications, one addition and one square root. Calculators are permitted throughout, and a student who states what a cosine of 0.96 means has met target 3 even if a partner did the arithmetic. See Differentiation & Access for the pre-computed magnitude scaffold.

Spiral — where this comes from and where it goes

• 6-8 Unit 1 (back-reference): transistors and logic gates established that everything is ultimately bits. This unit is the next honest question — given only numbers, how do you encode meaning?
• 6-8 Unit 2 (back-reference): you learned that a model needs features. An embedding is what happens when the machine learns its own features instead of being handed them.
• 6-8 Unit 4 (back-reference): you found bias in training data. Here you can point at exactly where it lives — as a direction in a vector space.
• 9-12 Unit 2 (forward): those embedding numbers are not given; they are trained, by gradient descent, which is the next unit.
• 9-12 Unit 3 (forward): attention computes weighted sums of these vectors using dot products — the same operation students did by hand today.
• 9-12 Unit 4 (forward): an LLM predicts the next token ID. Today's tokenizer beat is the reason that sentence will make sense.
• 9-12 Unit 8 (forward): retrieval-augmented generation finds relevant documents with cosine similarity over embeddings — Mat D, at production scale.
• 9-12 Unit 9 (forward): today's closing discussion becomes formal harm analysis and accountability for deployed systems.

Printable Materials — Representation: Numbers, Vectors, Embeddings

How to use: print one set per pair; Mats B, C and E are consumable, the card sets are reusable on cardstock. All arithmetic on these sheets is 2-D integer arithmetic with clean square roots — a basic calculator is sufficient and no device or internet access is required. Every card carries a label as well as a symbol, so nothing depends on colour.

Mat A — The representation pipeline · modelled by the teacher

Fill this in as the class works through the three encodings. The last column is the one that matters: what got thrown away.

Input	The rule that turns it into numbers	How many numbers	What is not in the numbers
The sentence I love unbelievable cats	Tokenize → look up each token's ID
A 224 × 224 colour photo		224 × 224 × 3 = ________
1 second of speech	Sample air pressure 44,100 times

Tokenizing practice. Split each string into plausible tokens and assign each a made-up ID from your own vocabulary table. (IDs are arbitrary labels — the point is that the same piece always gets the same number.)

String	Your tokens	Your IDs
cats
unbelievable
The cat sat.
antidisestablishmentarianism

Question to answer on the mat: why does a tokenizer keep the whole but split antidisestablishmentarianism into pieces?

Mat B — One-hot grid · the dead end

Vocabulary (5 words). Write a 1 in exactly one column per row.

Word	cat	dog	kitten	car	truck
cat
dog
kitten
car
truck

Now compute dot products. The dot product of two vectors is the sum of the products of matching positions.

Pair	Dot product	Are these two words related in real life?
cat · kitten
cat · truck
car · truck
dog · cat
Any two different words you choose: ________

Write the conclusion in your own words: In a one-hot representation, the similarity between any two different words is always ________, which means ________________________________.

Second problem — size. A real vocabulary has about 50,000 tokens.

	One-hot
Numbers stored per word	________
How many of them are 0	________
Percentage that carry no information	________ %

Mat C — The 2-D meaning space · plot these

Read this first. Real embeddings have hundreds of dimensions and no single dimension means anything on its own. This two-dimensional space with readable axes is a teaching model so you can see the geometry. Do not carry the readable axes forward as a belief about real models.

Word	Vector (x, y)
cat	(3, 4)
dog	(4, 3)
kitten	(6, 8)
car	(−4, 3)
truck	(−3, 4)

Plot all five as arrows from the origin on graph paper. Then answer:

1. Which words form a cluster? ________________________
2. Which arrow is longest? ________________________
3. cat and kitten point almost exactly the same way but one is much longer. Write one sentence guessing what length might be tracking, if it is not meaning. ________________________

Mat D — Cosine similarity worksheet

              a · b                    a₁b₁ + a₂b₂
cos(a, b) = ───────────   where  |a| = √(a₁² + a₂²)
             |a| × |b|

Step 1 — magnitudes. (All five are whole numbers. If you get a decimal, check your arithmetic.)

Word	Vector	a₁² + a₂²	|a|
cat	(3, 4)
dog	(4, 3)
kitten	(6, 8)
car	(−4, 3)
truck	(−3, 4)

Step 2 — the similarity table. Round to two decimal places.

Pair	Dot product	|a| × |b|	Cosine	Meaning (very similar / somewhat / unrelated / opposite)
cat, dog
cat, kitten
cat, car
cat, truck
dog, car
car, truck

Step 3 — the question that matters. The straight-line distance from cat to kitten is 5 units — the same as the distance from cat to the origin. Their cosine similarity is 1.00.

What does cosine similarity deliberately ignore, and why is ignoring it a good idea when you are measuring meaning?

Card set E — Analogy cards (cut apart) · one set per pair

Lay the four word-cards out, do the arithmetic on the strip, then answer the two footnote questions. Do not skip the footnote questions — they are the actual point of this card set.

👤 man = (2, 1)	👤 woman = (2, 3)
👑 king = (6, 1)	👑 queen = (6, 3)

Arithmetic strip:

Step	x	y
king
− man
+ woman
= result
Which word is the result closest to?

Footnote questions (required):

1. This worked exactly. In a real embedding the result is a point near the answer, not on it. Why do you think this toy version came out perfect? ________________________
2. When a real system does this, it excludes the three input words from the list of possible answers. What would probably be returned if it didn't? ________________________
3. Finish the sentence honestly: "This demo shows that ____________________. It does not show that ____________________."

Extra relations to try (build your own vectors so they work, then say why that's a cheat):

🌍 Paris : France :: Tokyo : ?	🗣️ walk : walked :: run : ?	🔢 dog : dogs :: mouse : ?

Card set F — Bias audit cards (cut apart) · one per pair

Each card names a corpus and a pair of words. Predict where a co-occurrence-trained embedding would place them relative to each other, and name who wrote the text. These are predictions about text, not claims about people.

Card	The corpus it learned from	Predict: which words land near each other, and why?
📰 Occupations	Decades of newspaper archives	nurse · surgeon · he · she
🩺 Medical notes	Hospital records from one country	patient · pain · symptom words in one language only
💬 Forum text	Public internet forums, 2005-2015	slang, insults, and the names of groups of people
📚 Books	Novels published before 1950	doctor · secretary · scientist · housewife
🌐 Web crawl	Whatever is most linked-to online	languages with billions of pages vs. languages with thousands
⚖️ Court filings	One jurisdiction's legal records	neighbourhood names · risk · arrest

Every card ends with the same three questions:

1. Who wrote this text, and who is missing from it?
2. If a hiring tool, a search engine or a chatbot were built on this embedding, what would go wrong — and for whom?
3. Would deleting the words he and she from the corpus fix it? Why or why not?

---

Teacher answer key (do not print for students)

Mat A. Text: 6 tokens (I love un believ able cats) → 6 IDs; what's thrown away — punctuation nuance, tone, who said it, and any meaning not recoverable from token order. Image: 150,528 numbers; what's thrown away — everything above the pixel level, including the fact that there is a cat. Audio: 44,100 numbers per second; what's thrown away — anything happening faster than the sampling rate can capture, plus the identity of the speaker as such. Tokenizer question: a fixed-size vocabulary cannot contain every word in a language, and it must still handle words it has never seen. Common words earn their own slot because they appear constantly; rare and long words are built from reusable pieces. Accept any answer containing "the vocabulary is a fixed size."

Mat B. Every dot product between two different one-hot vectors is 0, including cat · kitten and car · truck. (A word with itself is 1.) Conclusion: the representation encodes identity only — it can tell you whether two words are the same word and nothing else. Size: 50,000 numbers per word; 49,999 of them are 0; 99.998% carry no information.

Mat C. (1) Two clusters — cat, dog, kitten and car, truck. (2) kitten, magnitude 10. (3) Accept any reasonable guess; the answer worth surfacing is that in real embeddings magnitude tends to track frequency or the strength of the training signal, not meaning. Do not mark down a student who says "how much the model has seen the word" — that is essentially correct.

Mat D — Step 1 magnitudes. cat 25 → 5 · dog 25 → 5 · kitten 100 → 10 · car 25 → 5 · truck 25 → 5.

Mat D — Step 2 table (all exact):

Pair	Dot	|a|×|b|	Cosine	Meaning
cat, dog	24	25	0.96	very similar
cat, kitten	50	50	1.00	identical direction
cat, car	0	25	0.00	unrelated (perpendicular)
cat, truck	7	25	0.28	somewhat / weakly related
dog, car	−7	25	−0.28	pointing apart
car, truck	24	25	0.96	very similar

Step 3. Cosine ignores magnitude and measures only direction (angle). That is desirable because in a real embedding the magnitude largely reflects how often a token appears rather than what it means — two documents or words about the same topic should score as similar whether one is long and one is short.

Card set E. king − man + woman = (6, 1) − (2, 1) + (2, 3) = (6, 3) = queen, exactly. Footnotes: (1) because the space was constructed with a "royalty" axis and a "gendered" axis, so the relation is exactly linear by design; real embeddings are only approximately linear in this way. (2) most often king itself — the result lands nearest its own largest input, which is precisely why the exclusion rule exists and precisely why the demo is less impressive than it looks. (3) Model answer: "This shows there is linear structure in the space that partly lines up with some real relationships. It does not show the model understands kings, gender or royalty." On the extra relations: the intended realisation is that students can always invent vectors that make an analogy work — which is why the interesting question is never "does the arithmetic work?" but "did anyone choose the axes, or did they fall out of the data?"

Card set F. There is no single right answer; the evidence is the reasoning. Look for:

• The mechanism named correctly — the embedding places words near each other because they co-occur in the text, not because anyone believes anything.
• An awareness of who is absent from the corpus, not only who is present. Under-representation is the harder and more common problem.
• Question 3 is the one to press on. Deleting he and she does not work: the association is carried by hundreds of correlated words and contexts, not by two pronouns. This is the same finding as the "subtract a gender direction" result in the lesson — the association survives the edit and simply becomes harder to detect. If a pair answers "yes, that would fix it", that is the highest value re-teach in the unit.
• Anyone claiming the algorithm is "biased on purpose" should be redirected to the mechanism: the arithmetic did exactly what it was asked to. The corpus is the input, and people wrote it.

Vocabulary — Representation: Numbers, Vectors, Embeddings

Eight terms students will use to reason and compute, not recite. Every one of them reappears in Units 2, 3, 4 and 8 of this band, so post them and leave them up for the year. Push for a sentence that makes a claim you could check, not a definition.

Word	Meaning	Say it in a sentence
representation	The rule that turns a real thing into numbers a computer can hold.	"The representation decided what the model could learn — the model never saw the word."
token	A piece of text in the model's fixed vocabulary; roughly a word, sometimes a fragment.	"unbelievable came apart into three tokens because it's rare."
token ID	The integer that labels a token. Arbitrary — it is an index, not a quantity.	"Token 4092 isn't twice token 2046; IDs are labels, so the model looks up a vector instead of doing arithmetic on them."
vector	An ordered list of numbers; also a point, or an arrow from the origin, in a space.	"Each word is a vector, so 'close in meaning' becomes something I can measure."
one-hot vector	A vector that is all 0s with a single 1 marking which item it is.	"Every one-hot pair has dot product 0, so the representation can only tell me same word or not."
embedding	A dense learned vector, usually hundreds of dimensions, that places an item by meaning.	"In the embedding, cat and kitten landed near each other because they get used in the same sentences."
cosine similarity	The cosine of the angle between two vectors: +1 same direction, 0 unrelated, −1 opposite.	"cat and car scored 0.00 cosine similarity — perpendicular, so unrelated."
co-occurrence	How often two things appear near each other in the training text; the signal embeddings are learned from.	"The bias came from co-occurrence in the corpus, not from a mistake in the arithmetic."

Anchor idea

Keep returning to the sentence the whole unit turns on: the model never sees the word. Everything downstream follows from it. Training (Unit 2) adjusts these numbers; attention (Unit 3) takes dot products between them; an LLM (Unit 4) predicts the next ID; retrieval (Unit 8) ranks documents by cosine similarity. When a student asks "but does it understand?", the honest answer starts here: it has positions and angles, and every impressive thing it does is built from those.

The paired question to keep next to it: "Who wrote the text these positions were learned from?" The first question tells you what the machine is doing. The second tells you why the answers look the way they do.

Spiral note (for teachers)

These words tighten as the band progresses rather than being replaced. Vector and dot product become the mechanics of attention in Unit 3 — students who computed them by hand today will read the attention formula as arithmetic they already know. Embedding stops being a lookup table and becomes a contextual embedding in Unit 3, where the same token gets a different vector depending on the sentence around it; teaching the static version cleanly now is what makes that upgrade feel like news instead of a contradiction. Co-occurrence becomes pretraining in Unit 4 and the source of population-scale harm in Unit 9.

Back the other way: 6-8 Unit 1 established that everything is bits, and 6-8 Unit 2 established that models need features. The one-word upgrade this unit delivers is that the machine now learns its own features instead of being handed them — which is exactly what an embedding is.

One usage note worth holding: encourage "the vectors are close" and discourage "the model knows they're related." The first is something a student can compute and defend. The second smuggles in a claim about understanding that nothing in this unit — or the next three — actually supports.

Differentiation & Access — Representation: Numbers, Vectors, Embeddings

Designed against Universal Design for Learning. The one real access barrier in this unit is not the concept — it is the arithmetic, which can silently gate a student who understands the idea perfectly. Every scaffold below exists to keep the computation from becoming the assessment.

The rule that governs this whole file: interpreting a cosine value is the learning target. Producing it is the vehicle. A student who says "0.96 means those two point almost the same way, so they're used in similar sentences" has met target 3 whether or not they turned the handle.

Support (emerging learners & IEP)

• Pre-compute the magnitudes. Hand out Mat D with the |a| column already filled (5, 5, 10, 5, 5). The square root is where most students stall, and it is not what is being taught. Add the magnitudes back in later if it's useful.
• Give the dot product a fixed layout rather than a formula: two boxes to multiply, two boxes to multiply, one box to add. Many students who cannot parse Σaᵢbᵢ can execute the boxes flawlessly.
• Do one-hot first and slowly. It is the confidence beat of the unit — every student can build a one-hot vector, and the dot products are all 0, so nobody gets a wrong answer. Bank that before anything harder.
• Sentence frames for every interpretation: "A cosine of ____ means these two words point ____________, which tells me ____________." And for the closure: "These vectors came from ____________, so if the text mostly said ____________, then ____________."
• Reduce Mat D from six rows to three — cat/dog (0.96), cat/car (0.00), cat/kitten (1.00). Those three carry the entire concept; the other three are practice.
• Offer a pre-plotted Mat C. Plotting from coordinates is a separate skill from reading geometry. Let students who need it start from the finished picture.
• A calculator is standard equipment here, not an accommodation. Say so aloud to the class so nobody reads it as a signal.

English learners

• This unit is unusually friendly to multilingual students — the core content is numeric and the geometry is visual. Protect that advantage by keeping the arithmetic language-light: symbols and layout on the mats, explanation in words only where it is the target.
• Post the eight vocabulary terms with a picture next to each: a grid of 0s with one 1, two arrows at a small angle, two arrows at a right angle, a waveform with sample dots. The visual carries the term.
• Cognates help here. vector · vector, representation · representación, similarity · similitud, token · token. Name them explicitly; they are free.
• The tokenizer beat is a genuine expertise opportunity. Ask students who speak a language with long compound words, non-Latin script, or rich inflection what a subword tokenizer would have to do with their language — and then land the honest answer: models tokenize some languages far less efficiently than English, because the vocabulary was built mostly from English-heavy text. That is a real, verifiable observation, it belongs to them, and it is the sharpest possible entry into the bias discussion.
• Allow the bias-audit reasoning in the home language, then echo the English term back. Accept a spoken answer as full evidence.

Extension (advanced learners — classroom-anchored)

• Break the analogy. Give students free rein to find relations where a − b + c does not work, and to characterise which kinds fail. This is real research territory and there is no answer key — the honest observation is that some relation types travel linearly through the space and others simply do not.
• Prove it by hand. Show that for any non-zero scalar k, cos(ka, b) = cos(a, b). Three lines of algebra, and it is exactly the cat/kitten result generalised.
• The one-hot lookup insight. Ask what happens when you multiply a one-hot row vector by a matrix. (You select a single row.) Then ask what that means about the relationship between one-hot vectors and an embedding table. Students who reach "the embedding layer is a lookup table, and one-hot is the mathematical description of the lookup" have found something real and are ready for Unit 2.
• Curse of dimensionality, gently. In very high-dimensional spaces, randomly chosen vectors are nearly always close to perpendicular. Have students test it with random 2-D, then 3-D, then 10-D vectors and watch the cosines drift toward 0. This is why cosine is used instead of raw distance, and it is a genuinely surprising result.
• Debiasing critique. Give the strong version of the argument: you can identify a "gender direction" and project it out of every vector. Then give the finding: the biased words remain clustered together afterwards. Ask them to argue what that implies about measuring fairness by looking at a single dimension. Links directly forward to 9-12 Unit 9.
• Devices required for this one: implement cosine(a, b), then embed the whole vocabulary as a similarity matrix and sort each word's nearest neighbours. Spreadsheet is fine; no internet needed.

Access & accommodations

• Fine-motor: provide a pre-plotted Mat C and pre-cut card sets; allow answers to be pointed to, dictated or typed rather than written; use a large-format floor version of the coordinate grid with physical arrows if plotting is part of the goal.
• Vision / colour: every mat and card carries a label and a symbol, never colour alone. Provide a large-print set; the geometry is fully expressible verbally — "cat and dog point almost the same way; cat and car are at a right angle" — so a student using a screen reader or working non-visually can meet every target through the numbers alone. Offer a tactile version of Mat C (pins and string on foam board) if plotting is needed; the two clusters and the right angle are clearly felt.
• Hearing: the cosine formula, the vocabulary, the Hook question and all step directions go on the board in writing. No beat in this unit depends on hearing anything; the mats carry the content. Provide captioned or written versions of any video you add.
• Non-speaking / AAC: a student can demonstrate every target in writing or by selection — the interpretation column on Mat D is a four-way choice (very similar / somewhat / unrelated / opposite), and the bias-audit questions can be answered from a set of pre-written response cards. No target in this unit requires speech.
• Sensory / attention: offer the three-row version of Mat D rather than six; allow one card from set F rather than a rotation; the unit chunks cleanly at the step boundaries, so a student can stop after step 5 and still have met targets 1-4. Provide a quiet space for the performance task.
• Maths anxiety specifically: this is the accommodation most likely to be needed and least likely to be documented. Lead with "the arithmetic is deliberately small and the calculator is expected", give the pre-computed magnitude sheet without making it a request, and mark the performance task on the interpretations first. A student who has been told for years that they are "not a maths person" can absolutely hold this concept, and this unit is a good place to prove it to them.

A note on the bias discussion

This is the beat most likely to go sideways, in two directions.

• A class that dismisses it ("it's just data, it's not the computer's fault") is half right, and the half they have is worth affirming before you push. Yes — the arithmetic did exactly what it was asked. The question is what happens next, when someone builds a hiring filter on top of it and the geometry does the deciding at a scale no person reviews.
• A class that personalises it needs steering back to mechanism, quickly and without drama. Keep the discussion on corpora and co-occurrence, use the printed cards rather than open-ended prompts, and do not let a student's own experience become the class example. If someone raises something personal, acknowledge it briefly and honestly, then return to the text.

Never run open-ended "type identity words into an embedding tool and see what it says" as a class activity. It teaches nothing the structured cards do not, and it puts students in the position of watching a machine be publicly wrong about people like them.

Representation: Numbers, Vectors, Embeddings — Assessment

This unit is assessed through observation during the mats plus a short computational performance task. The computation is deliberately small; what is being assessed is whether a student can say what the number means, not whether they can do arithmetic quickly. Calculators are permitted throughout, and a student who states the interpretation correctly while a partner computes has still met target 3.

Learning targets

A student who has met this unit can:

1. Describe how text, an image or a sound becomes numbers, and name at least one thing the encoding throws away.
2. Build a one-hot vector and use the dot product to show that it cannot represent similarity.
3. Compute cosine similarity for a pair of 2-D vectors and interpret the result.
4. Explain that cosine measures direction, not magnitude, and why that is the right choice.
5. Perform analogy arithmetic on vectors and state a real limitation of the result.
6. Explain that embeddings are learned from co-occurrence in text, and identify where bias in an embedding comes from.

Observational checklist (mark per student or per pair during the mats)

Look-for	Target	Not yet	Developing	Got it
Tokenizes a string and states that token IDs are arbitrary labels, not quantities	1	○	○	○
Computes 224 × 224 × 3 and answers "which number is the cat?" correctly (none)	1	○	○	○
Builds correct one-hot vectors for the 5-word vocabulary	2	○	○	○
States that every distinct one-hot pair has dot product 0 — and says why that is fatal	2	○	○	○
Computes at least three cosine values correctly	3	○	○	○
Interprets a cosine value in words (very similar / unrelated / opposite)	3	○	○	○
Explains the cat/kitten case: distance 5, cosine 1.00	4	○	○	○
Completes king − man + woman correctly and answers a footnote question	5	○	○	○
States the distributional hypothesis in their own words	6	○	○	○
On a Bias audit card, locates the cause in the corpus, not in the arithmetic	6	○	○	○

Performance task — "Design the representation" (15-20 min, individual)

Give the student a new four-word vocabulary and a fresh set of 2-D vectors they did not use during the mats:

Word	Vector
doctor	(8, 6)
nurse	(6, 8)
piano	(−6, 8)
violin	(−8, 6)

Then ask them to do all six parts, on paper:

1. Write the one-hot vector for nurse in this four-word vocabulary, and compute the dot product of doctor and nurse in the one-hot representation. Say what that number tells you about how related the two words are. (Target 2)
2. Compute |doctor| and |nurse|, then the cosine similarity of doctor and nurse. (Target 3)
3. Compute the cosine similarity of doctor and piano, and of piano and violin. Say in one sentence what the three numbers together tell you about this space. (Target 3)
4. Suppose nurse were replaced by (12, 16) — twice as long, same direction. Does the cosine similarity with doctor change? Show it, and explain what cosine ignores and why that is sensible. (Target 4)
5. Given doctor = (8, 6) and nurse = (6, 8), invent two more words and vectors so that doctor − X + Y lands exactly on nurse. Then state one honest limitation of using this kind of arithmetic as evidence that a model "understands" the relationship. (Target 5)
6. Nobody typed these vectors in — they were learned from text. In two or three sentences, explain how a system could arrive at them from ordinary writing, and name one way bias could enter this particular four-word space. (Target 6)

Answers: (1) nurse = [0, 1, 0, 0]; dot product with doctor = 0, which says the one-hot representation treats them as completely unrelated — as it does every pair. (2) |doctor| = |nurse| = 10; dot = 48 + 48 = 96; cosine = 96 / 100 = 0.96. (3) doctor · piano = −48 + 48 = 0.00; piano · violin = 48 + 48 = 96 → 0.96. Two clusters, perpendicular to each other. (4) No — dot becomes 192, magnitudes 10 × 20 = 200, cosine = 0.96 again. Cosine ignores magnitude and measures only direction. (5) Any pair with the correct difference, e.g. X = (2, 0), Y = (0, 2). Limitations: the student chose the vectors so it would work; real results land near rather than on the answer; the input words are excluded from the answer set; it works for some relation types and not others. (6) Look for the distributional hypothesis — words appearing in similar contexts get similar vectors — and, for bias, the observation that the text decides which words sit near doctor and which sit near nurse, so whatever the writers assumed becomes geometry.

• Meets: parts 1-4 computed correctly with correct interpretations; part 5 invents a working pair and names a genuine limitation; part 6 names co-occurrence as the mechanism and locates bias in the corpus.
• Approaching: arithmetic is correct but interpretations are restated rather than explained ("0.96 means they're similar" with nothing behind it); part 5 does the arithmetic but the limitation is vague ("it's not always right"); part 6 says "the data was biased" without saying how data becomes geometry.
• Reteach: treats cosine as a distance, or gets a non-zero one-hot dot product, or claims a dimension of the vector "means" doctor-ness → return to Mat B (dot products all zero) and the cat/kitten row of Mat D. Do not move to Unit 2 until the one-hot result is secure; gradient descent is much harder to motivate without it.

Exit ticket (2 min, whole class)

On a slip, each student writes:

"cat and kitten are 5 units apart but have cosine similarity 1.00. In one sentence: why is that a good property for a system that measures meaning?"

Quick-scan for the three failure modes worth re-teaching tomorrow:

• "Cosine is distance" → the whole point of the cat/kitten row was that the two disagree.
• "The dimensions mean things" → readable axes were a scaffold; say so again explicitly.
• "The model knows what a cat is" → return to the closure: nothing in the pipeline ever contained a cat.

A note on what is not assessed

Students are not assessed on arithmetic speed, on memorising the cosine formula (give it to them on every task — it is a reference, not a recall item), on knowing real embedding dimensions, or on being able to name specific models or tools. They are also not assessed on their opinions during the bias discussion; the evidence for target 6 is whether they can locate the mechanism — text → co-occurrence → geometry — not whether they reach any particular conclusion about a corpus.

Evidence to keep: one completed Mat D per pair (the cosine table is a clean artefact and maps to targets 3 and 4 at once), a photograph of the class Mat A pipeline, and the performance task papers. Keep no copies of anything a student wrote about their own experience during the bias discussion.

Family Letter — Representation: Numbers, Vectors, Embeddings

Dear family,

This unit is the foundation for everything else your student will study about AI this year, and the idea at the centre of it is simpler than it sounds: a computer cannot store a meaning. It can only store numbers. So before any AI system can translate a sentence or answer a question, somebody has to decide what numbers will stand for a word. That decision turns out to matter enormously.

Your student worked through the obvious first idea and watched it fail. If you give every word its own slot — a long list of zeros with a single one marking which word it is — the computer can tell words apart, but it has no way to know that cat and kitten are related and cat and truck are not. Every pair looks equally unrelated. So the class moved to the approach modern AI actually uses: give each word a list of a few hundred numbers, treat that list as a position in space, and let words that mean similar things sit near each other. Once meaning is a position, "how similar are these two words?" becomes something you can genuinely measure — and your student measured it, by hand, with a calculator.

There is a famous party trick in this area: with the right numbers, king minus man plus woman lands on queen. We taught it, because it is real and it is delightful. We also taught the part that usually gets left off — the answer is only approximately right, the method quietly excludes the obvious wrong answers, and it works far better for some kinds of relationships than others. Knowing what a demonstration leaves out is one of the most useful habits a student can build about AI, and this unit is a good, low-stakes place to build it.

Two things we handled carefully. First, nobody typed those numbers in. They are learned from enormous amounts of ordinary writing, on the principle that words used in similar sentences probably mean similar things. Which leads to the second thing: whatever patterns are in the writing end up in the numbers, including ones we would not choose. The class discussed this technically — the arithmetic was not wrong, it measured the text it was given — and kept the conversation about text and mechanism rather than about anybody in the room.

Try this together — no device needed, about ten minutes. Pick any four things from one category: four foods, four musical artists, four cars, four cities. Now each of you, separately, place them on a piece of paper as dots, putting things you think are similar close together. Compare your two maps. You will almost certainly disagree — and the interesting question is why: what were you each measuring? Price? Familiarity? Something you couldn't name? That is exactly the problem an AI system faces, and exactly why the text it learns from decides the answer. If your student is willing, ask them to explain how a computer measures the angle between two of those dots. If they can teach it to you, they have it.

A note on devices and data: the core of this unit is unplugged — printed sheets, graph paper and a basic calculator. One optional extension uses a spreadsheet or a small Python program on a school device. No student information was entered into any AI tool at any point, and nothing personal was collected.

Thank you for learning alongside us.

---

This letter is available in other languages — just ask your child's teacher. Esta carta está disponible en otros idiomas — pregunte al maestro de su hijo/a.

Take-Home — Meaning as a Position

You need: a pencil, a calculator with a square-root key, and a sheet of graph paper. No device and no internet. About 40 minutes.

The idea in one sentence. A computer cannot store a meaning — it can only store numbers. So every AI system starts with somebody deciding what numbers will stand for a word, and that decision sets the ceiling on everything the system can ever do.

---

Part 1 — Everything becomes numbers

Before a model reads anything, the thing gets converted. Fill in the third column.

Input	How it becomes numbers	What is not in those numbers?
The sentence I love unbelievable cats	A tokenizer cuts it into pieces (I love un believ able cats); each piece has an integer ID
A 224 × 224 colour photo	Each pixel is three numbers 0-255 (red, green, blue)
One second of speech	Air pressure measured 44,100 times per second

1.1 How many numbers are in that 224 × 224 colour photo? Show the multiplication.

1.2 Which one of those numbers is the cat in the picture? Explain your answer in one sentence.

1.3 A tokenizer keeps the whole but splits antidisestablishmentarianism into several pieces. Given that the vocabulary is a fixed size and language is not, why is that the sensible design?

Careful: a token ID is a label, not a quantity. Token 4092 is not "twice" token 2046. The model uses the ID to look up a list of numbers, and it is that list that carries the meaning.

---

Part 2 — The obvious idea, and why it dies

Here is the first thing anyone tries. Give every word in the vocabulary its own slot. That is a one-hot vector. Fill in the grid — one 1 per row, the rest 0.

Vocabulary: coffee, tea, espresso, bicycle, scooter

Word	coffee	tea	espresso	bicycle	scooter
coffee
tea
espresso
bicycle
scooter

The dot product of two vectors is the sum of the products of matching positions: a · b = a₁b₁ + a₂b₂ + … + aₙbₙ. Compute these.

Pair	Dot product	Are these two words related in real life?
coffee · espresso
coffee · bicycle
bicycle · scooter
any two different words you pick: ________

2.1 Complete the sentence: "In a one-hot representation, the similarity between any two different words is always ________, which means the representation can only tell me ________________________________."

2.2 A real vocabulary has about 50,000 tokens. In a one-hot representation, how many numbers are stored per word, how many of them are 0, and what percentage of the storage carries no information?

2.3 So what? Explain, in two sentences, why a model built on one-hot vectors would have to learn coffee and espresso completely separately, with nothing transferring between them.

---

Part 3 — Meaning becomes a position

The fix is to stop giving each word its own slot, and instead give every word a short list of numbers that places it somewhere. Words that mean similar things end up near each other. That is an embedding.

Real embeddings have hundreds of dimensions and no single dimension means anything on its own. The two-dimensional version below is a teaching model so you can see the geometry — don't carry the readable picture forward as a belief about real systems.

Word	Vector
coffee	(4, 3)
tea	(3, 4)
espresso	(8, 6)
bicycle	(−3, 4)
scooter	(−4, 3)

3.1 Plot all five as arrows from the origin on graph paper. Which words form a cluster?

3.2 Which arrow is longest? Write one sentence guessing what length might be tracking, if it is not meaning.

---

Part 4 — Measuring closeness

Cosine similarity measures the angle between two vectors and ignores how long they are:

              a · b                     a₁b₁ + a₂b₂
cos(a, b) = ───────────  =  ──────────────────────────────
             |a| × |b|      √(a₁²+a₂²) × √(b₁²+b₂²)

It runs from +1 (same direction) through 0 (perpendicular — unrelated) to −1 (opposite).

Step 1 — magnitudes. All five come out as whole numbers. If you get a decimal, check your work.

Word	Vector	a₁² + a₂²	|a|
coffee	(4, 3)
tea	(3, 4)
espresso	(8, 6)
bicycle	(−3, 4)
scooter	(−4, 3)

Step 2 — similarities. Round to two decimal places.

Pair	Dot product	|a| × |b|	Cosine	Very similar / somewhat / unrelated / opposite
coffee, tea
coffee, espresso
coffee, bicycle
coffee, scooter
bicycle, scooter

4.1 The straight-line distance from coffee to espresso is 5 units — they are genuinely far apart on your graph paper. Their cosine similarity is 1.00. What did cosine deliberately ignore?

4.2 Why is ignoring that a good idea when the thing you are measuring is meaning?

4.3 Here is that formula as code. Read it and answer: which line does the "ignore the length" part?

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
    return dot / mag

---

Part 5 — The famous trick, and the small print

If meanings are positions, then the difference between two positions might be a relationship. Take these four vectors:

man = (2, 1) · woman = (2, 3) · king = (6, 1) · queen = (6, 3)

5.1 Compute king − man + woman, coordinate by coordinate. Which word does the result land on?

5.2 That worked exactly. In a real embedding, the result is a point near the answer and the system returns the nearest word to it. Why do you think this version came out perfect? (Look at what the two axes are doing.)

5.3 When a real system does this, it excludes the three input words from the list of possible answers. If it didn't, the most common result would be king itself. Does knowing that make the demonstration more impressive or less? Defend your answer in two sentences.

5.4 Finish the sentence honestly: "This shows that ____________________. It does not show that ____________________."

---

Part 6 — Where the numbers came from

Nobody typed these vectors in. They are learned, from one stubborn observation: words that show up in similar contexts tend to mean similar things. Feed a system enormous amounts of ordinary writing, nudge each word's vector toward the words it appears near and away from the words it doesn't, and after billions of small adjustments the geometry falls out on its own. coffee and espresso end up close because people use them in the same sentences — never because anyone told the machine what coffee is.

6.1 If the vectors come from what people actually wrote, and a huge archive of writing mostly puts nurse near she and surgeon near he, what happens to the geometry? Answer in one sentence, using the word co-occurrence.

6.2 Someone says: "That's a bug in the algorithm." Are they right? Explain what the arithmetic actually did.

6.3 Someone else says: "Easy — just delete the words he and she from the training text." Would that work? Why or why not?

Essential question: If a machine only ever sees numbers, where does meaning actually live — and who put it there?

Talk about it with someone at home. Pick any four things in one category — four foods, four cities, four songs. Each of you, separately, place them as dots on paper with similar things close together. Then compare your maps. You will disagree, and the interesting question is why: what were you each measuring? That disagreement is exactly the problem an AI system faces — and it is why the text a model learns from decides the answer.

Practice Set — Representation: Numbers, Vectors, Embeddings

How to use: One copy per student, about 45 minutes. Parts 1–2 are the core skills, Parts 3–4 ask you to interpret and debug, and the Challenge is a design task. A calculator with a square-root key is allowed throughout. Round cosine values to two decimal places unless a question says otherwise. Answers are in the Answer Key.

Your reference card

• Dot product: a · b = a₁b₁ + a₂b₂ + … + aₙbₙ
• Length (magnitude): |a| = √(a₁² + a₂² + … + aₙ²)
• Cosine similarity: cos(a, b) = (a · b) / (|a| × |b|). It runs from +1 (same direction) through 0 (perpendicular, unrelated) to −1 (opposite).
• The 2-D vectors on this sheet are a teaching model. Real embeddings have hundreds of dimensions, and no single dimension means anything on its own.

Part 1 — Everything becomes numbers

1. Fill in the table.

Input	How many numbers? Show the multiplication.	Name one thing that is not in those numbers
A 64 × 64 colour photo of a dog (each pixel stores red, green and blue)
A 5-second voice memo, sampled 16,000 times per second

Which one of the photo's numbers is the dog? _______________________________________________

2. A tokenizer gives the token sun the ID 1500 and the token moon the ID 3000. Priya says, "So the model treats moon as twice as much as sun."

(a) What is wrong with Priya's claim? _______________________________________________

(b) What does the model actually do with a token ID? _______________________________________________

3. Vocabulary of five words: red, orange, blue, spoon, fork.

(a) Write the one-hot vectors.

Word	red	orange	blue	spoon	fork
orange
fork

(b) Compute the dot products.

Pair	Dot product
red · orange
spoon · fork
orange · orange

(c) Complete the sentence: "In a one-hot representation, the dot product of two different words is always ______, so the representation can only tell me ______________________________."

4. A real vocabulary has 32,000 tokens. In a one-hot representation:

(a) How many numbers are stored for each word? ____________

(b) How many of those numbers are 0? ____________

(c) What percentage of the numbers are 0? Round to three decimal places. ____________ %

Part 2 — Cosine similarity by hand

Use this teaching space for items 5–7.

Word	Vector
ocean	(12, 5)
sea	(4, 3)
lake	(12, 9)
mountain	(−5, 12)
hill	(−3, 4)

5. Find each magnitude. (All five are whole numbers. If you get a decimal, check your arithmetic.)

Word	Vector	a₁² + a₂²	|a|
ocean	(12, 5)
sea	(4, 3)
lake	(12, 9)
mountain	(−5, 12)
hill	(−3, 4)

6. Complete the similarity table.

Pair	Dot product	|a| × |b|	Cosine	Very similar / somewhat / unrelated / pointing apart
ocean, sea
sea, lake
ocean, mountain
ocean, hill
mountain, hill

7. Straight-line distance between two points is √((x₂ − x₁)² + (y₂ − y₁)²).

(a) Distance from sea to lake: ____________ Distance from sea to hill (two decimal places): ____________

(b) By straight-line distance, which word is closer to sea? ____________ By cosine similarity? ____________

(c) A search tool is supposed to find words with similar meaning. Which measurement should it use? What does that measurement deliberately ignore, and why is ignoring it a good idea?

---

---

Part 3 — Analogy arithmetic, and the small print

Use these vectors for items 8–10: cold = (1, 3) · colder = (4, 4) · warm = (2, 6) · warmer = (5, 7)

8. Compute colder − cold + warm, coordinate by coordinate.

Step	x	y
colder
− cold
+ warm
= result

Which word does the result land on? ____________

9. Real systems work in hundreds of dimensions, and the result almost never lands exactly on a word. The system ranks words by cosine similarity to the result and returns the top one. Here is a made-up ranking, for illustration only, for colder − cold + warm:

Rank	Word	Cosine with the result
1	warm	0.93
2	warmer	0.90
3	hotter	0.84
4	colder	0.79
5	heat	0.71

(a) With no extra rule, which word does the system return? ____________

(b) Standard practice excludes the three input words from the answer. Now which word is returned? ____________

(c) Use your arithmetic from item 8. What is result − warm? Use it to explain why the result sits so close to warm. _______________________________________________

10. A headline reads: "AI understands temperature: colder − cold + warm = warmer!" Write a two-sentence correction. The first sentence says what the arithmetic does show; the second says what it does not show, and why.

---

---

Part 4 — Find the bug, find the source

11. Jordan wrote this function.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag = sum(x * x for x in a) * sum(y * y for y in b)
    return dot / mag

(a) What does cosine((4, 3), (12, 9)) return? Round to three decimal places. ____________

(b) What should it return? How could you know that without doing any arithmetic?

---

(c) Which line has the bug? Write the corrected line. _______________________________________________

12. Sam's table says cos(mountain, hill) = 12.6. Without redoing any arithmetic, explain how you know that is wrong. Then find Sam's mistake. (Hint: 63 ÷ 5 = 12.6.)

---

13. Three sentences from a training corpus:

The band's trumpet player warmed up before the concert. The band's trombone player warmed up before the concert. The mechanic warmed up the engine before the race.

A system learns word vectors from co-occurrence. Which word ends up closer to trumpet — trombone or engine? Point to the evidence in the sentences. Did anyone tell the system what a trombone is?

---

14. A word embedding learned from decades of job advertisements places engineer near he and receptionist near she.

(a) Where did that pattern come from — the arithmetic, or the text? Answer in one sentence that uses the word co-occurrence. _______________________________________________

(b) A developer subtracts a "gender direction" from every vector and announces that the bias is gone. Why is that claim too strong? _______________________________________________

(c) A company uses this embedding to rank job applications. Who is accountable for the results — the vectors, or a person? Explain. _______________________________________________

Challenge — Design the representation

15. Four words: sunrise, morning, sunset, evening. Give each one a 2-D vector with whole-number coordinates so that all three conditions are true:

• cos(sunrise, morning) ≥ 0.90 and cos(sunset, evening) ≥ 0.90
• cos(morning, evening) is between −0.10 and 0.10
• morning − sunrise + sunset lands exactly on evening

Word	Your vector
sunrise
morning
sunset
evening

Show all three cosine values (two decimal places) and the analogy arithmetic. Then write one honest sentence explaining why your perfect analogy is not evidence that a trained model understands time of day.

---

---

Answer Key — Representation: Numbers, Vectors, Embeddings

How to use: Worked answers for the Practice Set, item by item. Rounding rule: cosine values and distances to two decimal places; the percentage in item 4 to three. Where more than one answer works, the key says what to accept. Constructed responses carry a Full credit note saying what the answer must contain. The test for every interpretation question is the same: can the student say what the number means, not just produce it?

Part 1 — Everything becomes numbers (Empower Student Success)

1.

Input	How many numbers?	Not in the numbers (sample answers)
64 × 64 colour photo	64 × 64 × 3 = 12,288	the fact that there is a dog; what the dog is doing; anything above the level of single pixel brightness
5-second voice memo	5 × 16,000 = 80,000	the meaning of the words; who is speaking, as a fact; tone, as an idea

Which number is the dog? None of them. The dog is a pattern spread across thousands of brightness values; no single number stands for it. Accept any reasonable "not in the numbers" answer that names something a person understands but a list of measurements does not state.

2. (a) Token IDs are arbitrary labels, not quantities. 3000 is not "twice" 1500 in any sense the model uses; the IDs could be swapped and nothing about meaning would change. (b) The model uses the ID to look up a vector (an embedding). The vector carries the information, not the ID. Accept "it's an index" or "a row number in a table of vectors."

3. (a) orange = [0, 1, 0, 0, 0] · fork = [0, 0, 0, 0, 1]

(b)

Pair	Dot product
red · orange	0
spoon · fork	0
orange · orange	1

(c) "…always 0, so the representation can only tell me whether two words are the same word — nothing about how similar they are."

Common mistake: giving spoon · fork a non-zero value because spoons and forks are related in real life. The one-hot representation cannot see that. That blindness is the point of the item.

4. (a) 32,000 (b) 31,999 (c) 31,999 ÷ 32,000 × 100 = 99.996875 → 99.997%

Part 2 — Cosine similarity by hand (Empower Student Success)

5.

Word	a₁² + a₂²	|a|
ocean	144 + 25 = 169	13
sea	16 + 9 = 25	5
lake	144 + 81 = 225	15
mountain	25 + 144 = 169	13
hill	9 + 16 = 25	5

6.

Pair	Dot product	|a| × |b|	Cosine	Meaning
ocean, sea	(12)(4) + (5)(3) = 63	13 × 5 = 65	63 ÷ 65 = 0.9692 → 0.97	very similar
sea, lake	(4)(12) + (3)(9) = 75	5 × 15 = 75	1.00	very similar — identical direction
ocean, mountain	(12)(−5) + (5)(12) = 0	13 × 13 = 169	0.00	unrelated (perpendicular)
ocean, hill	(12)(−3) + (5)(4) = −16	13 × 5 = 65	−16 ÷ 65 = −0.2462 → −0.25	pointing apart
mountain, hill	(−5)(−3) + (12)(4) = 63	13 × 5 = 65	0.97	very similar

Common mistake: writing (−5)(−3) as −15. A negative times a negative is positive, and getting it wrong turns mountain, hill from 0.97 into 0.51. Accept "somewhat unrelated" for the −0.25 row.

7. (a) sea to lake: √((12 − 4)² + (9 − 3)²) = √(64 + 36) = √100 = 10. sea to hill: √((−3 − 4)² + (4 − 3)²) = √(49 + 1) = √50 = 7.071 → 7.07.

(b) By distance, hill is closer (7.07 < 10). By cosine, lake is closer: cos(sea, lake) = 1.00, while cos(sea, hill) = (4)(−3) + (3)(4) = 0 → 0.00.

(c) Cosine similarity. It ignores length (magnitude) and measures only direction. That is the right choice for meaning because in real embeddings length tends to track something else, such as how often a word appears. Two words about the same thing should score as similar even when one vector is longer.

Full credit: names cosine, says it ignores length/magnitude, and gives a reason that ignoring length suits meaning. "Cosine is more accurate" with no reason is not enough.

Part 3 — Analogy arithmetic, and the small print (Empower Student Success)

8.

Step	x	y
colder	4	4
− cold	− 1	− 3
+ warm	+ 2	+ 6
= result	5	7

The result is (5, 7) = warmer, exactly.

9. (a) warm (0.93, the top of the list). (b) warmer. warm, cold and colder are excluded, and warmer (0.90) is the highest word left. (c) result − warm = colder − cold = (4 − 1, 4 − 3) = (3, 1). The result is just warm plus the small step from cold to colder. That step has length √10 ≈ 3.16, about half the length of warm itself (√40 ≈ 6.32), so the result stays close to its own largest input. This is exactly why the exclusion rule exists, and why it makes the demonstration less impressive than it first looks. Accept any answer that shows the result equals warm plus the difference between colder and cold.

10. Model answer: "The arithmetic shows the space has structure: the step from cold to colder points the same way as the step from warm to warmer. It does not show that the model understands temperature, because the vectors came from word co-occurrence (or, here, were chosen), real results land near the answer rather than on it, the input words are excluded from the answer, and the trick fails for many other relations."

Full credit: the first sentence keeps a defensible claim (a pattern or linear structure exists in the space); the second removes the claim of understanding and gives at least one real reason from the list above.

Part 4 — Find the bug, find the source (Empower Student Success, Augment Not Replace)

11. (a) dot = 48 + 27 = 75; mag = 25 × 225 = 5,625; 75 ÷ 5,625 = 0.01333… → 0.013. (b) 1.00. (12, 9) is exactly 3 × (4, 3), so the two vectors point the same way, and cosine must be 1 no matter how long they are. (c) The mag line is missing the square roots: mag = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)) Accept ** 0.5 in place of math.sqrt. Note the clue in the original: math is imported but never used.

12. Cosine similarity can never be greater than 1 or less than −1, so 12.6 is impossible. Sam divided the dot product (63) by |hill| = 5 only, instead of by |mountain| × |hill| = 13 × 5 = 65. Correct value: 63 ÷ 65 → 0.97.

13. trombone. It appears in the same context as trumpet: "The band's ___ player warmed up before the concert." engine shares only "warmed up… before the". Nobody told the system what a trombone is; it only counted which words appear near which. Words used in similar contexts end up with similar vectors. Accept any answer that points to the shared surrounding words as the evidence.

14. (a) From the text. In those advertisements engineer co-occurred with he far more often than with she, so the arithmetic placed them close together. It measured the corpus correctly.

(b) The association is carried by many correlated words and contexts, not by a single direction. Researchers have shown that after a "gender direction" is subtracted, the biased words are still clustered together. The edit makes the association harder to see; it does not remove it. Fixing the numbers is not the same as fixing what they represent.

(c) A person, meaning the people who chose to use this embedding for hiring and who act on its rankings. Vectors cannot be accountable. People chose the corpus, the tool, and how its output is used.

Full credit: (a) locates the cause in the corpus through co-occurrence, not in the arithmetic; (b) says the association survives the edit or becomes harder to detect; (c) names people as accountable, with a reason. Do not mark down an answer for the opinion it reaches, only for a missing mechanism.

Challenge — Design the representation (Empower Student Success)

15. Answers vary. One correct version:

Word	Vector
sunrise	(6, 2)
morning	(5, 0)
sunset	(1, 4)
evening	(0, 2)

• cos(sunrise, morning) = (30 + 0) ÷ (√40 × 5) = 30 ÷ 31.62 → 0.95 ✓
• cos(sunset, evening) = (0 + 8) ÷ (√17 × 2) = 8 ÷ 8.25 → 0.97 ✓
• cos(morning, evening) = (0 + 0) ÷ (5 × 2) → 0.00 ✓
• morning − sunrise + sunset = (5 − 6 + 1, 0 − 2 + 4) = (0, 2) = evening ✓

A fast design strategy worth praising: pick sunrise, morning and sunset first, compute evening = morning − sunrise + sunset, and only then check the cosines.

Honest sentence (model answer): "I chose these coordinates so the analogy would work, so it landing exactly tells you about my design, not about understanding; a trained model's vectors come from co-occurrence in text, and its analogies only land near the answer."

Full credit: four whole-number vectors; all three cosine values computed correctly (two decimal places) for the student's own vectors and meeting every condition; the analogy arithmetic shown landing exactly on evening; the limitation names that the vectors were chosen, or another genuine limitation. If a condition fails but the student's arithmetic correctly shows it failing, give credit for the computation and ask for one revision.