INV-06 · The analyst that never invents a number

Build an investment-memo drafter that cannot invent a number. Every number it writes must cite its source, and every number must match the calculator, or the build fails. Can you prove it, on twenty test cases you wrote to break it?

The point is not a clever drafter. The point is the checking code around it: the part that would stop a junior analyst — human or model — from putting a wrong number in front of an investment committee.

CASE-ACME and the student's own INV-02 or L-01 to L-04 results. Python or JavaScript on a school computer. Stub provider only: no AI model, no API key, no network call, and no credentials anywhere in the code or its configuration. The student needs:

• A ground-truth table — every number the memo is allowed to use, each with its source: a case-file line, or a logged calculator run (which calculator, which inputs, which output). Built by the student from their own calculator work, not by the drafter.
• A memo schema — a structured form, not free text: memo sections, each a list of claims; each claim has its text, any number it states, the unit, and a citation to a ground-truth row.
• A stub model — a function that returns scripted memo drafts written by the student. Most are correct. Some are deliberately wrong, so the checks have something real to catch.
• Three checks, in order: parse (is it valid structured output?), validate (does every claim match the schema, and does every number carry a citation?), reconcile (does every cited number match its ground-truth row, within a stated tolerance?). One repair retry is allowed after a parse or validate failure; never after a reconcile failure, because a wrong number is not a formatting problem.
• A conservation check borrowed from the calculator's own tests: if the memo states a payout for every holder at an exit, the payouts must sum to the exit value.

A drafter that writes numbers in prose that the schema never sees. Citations that point at a row but are never compared to it. A tolerance so loose that a wrong number passes. An eval set made only of easy cases. A repair loop that "fixes" a wrong number by asking again until it changes. Code that reads an environment variable "for later, when we use a real model" — the build has no place for credentials, and that is the design.

A packet with five pieces:

1. The ground-truth table, with every row's source.
2. The schema and the three checks, with the tolerance stated and justified (to the cent for dollar amounts and to two decimal places for percentages are reasonable defaults).
3. The 20-case eval set — twenty scripted stub drafts, each labeled with what it should do. At least: five fully correct drafts; three with an invented number (a figure in no ground-truth row); three with a correct number cited to the wrong row; two with a units error ($M versus $, or 0.25 versus 25%); two with a number in prose but missing from its claim's number field; two with invalid structure that one repair fixes; one with invalid structure that repair cannot fix; one whose payouts do not sum to the exit; and one whose rounding falls just outside the tolerance.
4. The eval report — a table of all twenty cases: expected result, actual result, which check caught it. Every invented-number case must be caught. Anything less is a failed build, reported as one.
5. The memo — one investment-memo page on Acme's Series B drafted by the stub and passed through all three checks, with every number cited, and a note at the bottom: "Every number on this page was reconciled against a calculator before publication. This is a class exercise on a fictional company. It is not legal, tax, or investment advice."

CT: designing checks that catch failures you have not seen yet. CR: writing adversarial test cases against your own code. CO: a peer writes three extra wrong drafts without seeing the eval set; the report says whether the checks caught them. CM: the eval report, readable by someone who will never read the code. CZ: the investment committee, and everyone downstream of it, who trusts a number because it appeared in a memo.

A software engineer, data scientist, or fund analyst reviews the eval set at Checkpoint 1 — before the checks are finished, since an eval written after the code tends to test what the code already does — and the eval report at Checkpoint 2. Standing instruction: reject any build where an invented number passes, and any code that reads a key, password, or environment variable. School-supervised, both times.

R1: one memo, one company, one ground-truth table. R2: every number cited and reconciled; eval re-runnable. R3: the calculator is the baseline; the drafter adds formatting and nothing else it can prove. R4: the twenty cases cover the named failure types; the tolerance is justified. R5: the eval report is clear to a non-programmer. R6: names what the drafter refuses to write — any number not in the ground-truth table.

(a) Twenty passing cases, all easy, so the report shows 100% and proves nothing. (b) A careful reconcile step that the memo's prose bypasses, so an invented number reaches the page in a sentence instead of a field.

• Week 1–2: Ground-truth table complete and traced; schema drafted; the 20-case eval set written and mentor-reviewed before the checks are built.
• Week 3–5: Stub, parse, validate, and reconcile checks built; first full eval run; failures fixed in the checks, never by editing the eval cases.
• Week 6: Peer's three extra wrong drafts run; eval report and memo final; teacher confirms the code reads no credentials and makes no network call.

Lane A immediately (computer science, statistics, or finance capstone). Pairs with Applied ML. Lane B only where the school's work-based-learning coordinator owns the record. No verified credit claim.