Applied ML Systems — Capstone Brief Bank

Track: Applied ML Systems · data science pathway Built for: Henrico County Public Schools, Center for Information Technology at Deep Run Briefs: 25 (DS-01 … DS-25) Cost: Free. Keep it, copy it, remix it. No agreement, no purchase order, no data privacy agreement — this is a document, not a system, and it moves no student data.

⚠️ Read before you publish or assign

Every dataset URL in this bank is marked UNVERIFIED. They were written from knowledge of these sources, not confirmed live at time of writing. Public data moves: agencies re-platform, portals retire, and file paths change without redirects.

Before you hand any brief to a student, check its source. Run:

node scripts/check-brief-datasets.mjs

against datasets.json in this folder. It reports every URL as live, moved, or dead. A brief whose source has moved is still a good brief — swap the URL and keep going.

How to use this bank

Students choose; teachers do not invent. That is the whole point. A teacher's job here is to run the checkpoints and score the work, not to design twenty-five projects a year.

Each brief is written to fit an existing culminating requirement — a center senior capstone, AP Research, PLTW Capstone, or an IB Extended Essay. None of them require a new course, a new period, or a board vote.

These briefs do not produce verified credit. No AI or computer science course in Virginia does. Where a brief connects to High-Quality Work-Based Learning, it does so through an Internship or an Entrepreneurship experience — the two HQWBL types that fit this work. Mentorship is not an HQWBL type, and neither are job shadowing, externships, co-ops, or service learning.

The mentor role is advisory. Always.

Every brief names a mentor role. In every case that role is advisory only:

The mentor reviews student work at two scheduled checkpoints and gives feedback.
The school supervises. Every interaction happens in a school-supervised setting, under your division's policies, during your hours.
Never one-to-one unsupervised. Not in person, not by DM, not by private call.
The mentor never handles student data, never receives student contact details, and never takes custody of a student's work product.
The mentor is sourced and screened by your division, through the volunteer pipeline it already operates. Schoolhouse does not recruit, screen, match, or place mentors, and cannot accept that responsibility on your behalf.

If you cannot staff a mentor, run the brief without one. Every brief in this bank is completable with a teacher and a rubric. The mentor improves the work; it is not a dependency.

The six-dimension rubric

One instrument, used across all 25 briefs, so a director can defend a score and a student can see the target from day one. Score each dimension 1–4 (1 = not yet, 2 = developing, 3 = proficient, 4 = exemplary). Each brief adds a short calibration saying what a 4 looks like for that problem specifically.

#	Dimension	The question it answers
R1	Problem framing	Is the question actually answerable with the data at hand, and did the student narrow it themselves?
R2	Data provenance & handling	Where did the data come from, what did the student change, and can someone else reproduce it?
R3	Baseline & model	Is there an honest baseline, and does the model beat it for a reason the student can name?
R4	Evaluation & error analysis	Did the student measure the thing that matters, look at what failed, and quantify uncertainty?
R5	Communication & model card	Can a non-specialist read the result and correctly understand what it does and does not support?
R6	Ethics, limits & refusal	Who is affected, how does this fail in the world, and what did the student decline to claim?

R6 is not a participation dimension. A student who ships an accurate model and cannot say who it hurts has not finished the assignment. A student who identifies a harm and narrows the claim in response is demonstrating the highest thing this rubric measures.

The five C's

Virginia requires every graduate to demonstrate critical thinking, creative thinking, collaboration, communication, and citizenship. Each brief maps to all five, and each maps them differently — the evidence file should quote the specific artifact, not the category. Use evidence-template.md conventions: artifact, date, what it shows.

Shorthand used below: CT critical thinking · CR creative thinking · CO collaboration · CM communication · CZ citizenship.

Two standing failure modes

Before the per-brief list, two that apply to all 25 and cause more failed capstones than anything technical:

Scope inflation in week two. The student picks a bigger question than the data can answer, and spends the term collecting instead of analyzing. Checkpoint 1 exists to catch this. Cut scope, do not extend the timeline.
The accuracy trap. The student optimizes one number, hits it, and has nothing to say. Every brief below names the metric that actually matters — it is rarely accuracy.

The briefs

DS-01 · Predicting when the bus is actually coming

The question. GRTC publishes a schedule and a real-time feed. How far apart are they, and can you predict the gap for a given route, stop, and time of day better than assuming the bus is on time?

Data. GRTC GTFS static and GTFS-Realtime feeds (UNVERIFIED — https://ridegrtc.com/developer-resources/). Supplement with NOAA hourly weather for the Richmond area (UNVERIFIED — https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily). Real-time feeds must be collected over time — a student starting in week one has eight weeks of data by the deadline; a student starting in week six has nothing.

Expected failure modes. Realtime feeds drop out for hours and the gaps look like on-time performance. Trip IDs change between schedule versions and silently break joins. Rush-hour and midday behave like different problems. Predicting delay in minutes looks easy because most buses are near-zero — the model learns to say "on time" and scores well.

Done looks like. A model that predicts arrival delay at a named stop, evaluated against the "assume the schedule" baseline, with error broken out by time of day and route — plus an honest statement of how many hours of feed were actually captured.

Five C's. CT: choosing an error metric that punishes the costly direction (early buses strand riders; late buses annoy them). CR: designing the collection pipeline before there is data to analyze. CO: comparing capture logs with a peer to find dropouts. CM: a rider- facing explanation of what "3 minutes late, usually" means. CZ: who is harmed by an unreliable bus, and does the error concentrate on particular routes.

Mentor role (advisory). A transit planner, data engineer, or logistics analyst reviews the collection design at Checkpoint 1 and the error analysis at Checkpoint 2. School- supervised, both times.

Rubric calibration. R1: the question is narrowed to specific routes and hours, with a stated reason. R2: collection gaps are logged and reported, not hidden. R3: beats the schedule baseline, and the student can say which conditions drive the gain. R4: error split by hour and route, with the early/late asymmetry addressed. R5: a rider could read it. R6: names which riders bear the worst predictions.

Two ways this goes wrong. (a) The student starts collecting in week six, has four days of data, and pads with schedule-only rows — the model learns the schedule and reports excellent accuracy. (b) The student reports mean absolute error of 2 minutes and calls it a success, without noticing every large error is on one late-night route.

DS-02 · Is the property assessment fair across the county?

The question. Henrico publishes assessed values and sale prices. Assessment regressivity — where lower-value homes are assessed at a higher fraction of market value than higher-value homes — is measurable. Is it present here?

Data. Henrico County open GIS and real estate assessment data (UNVERIFIED — https://data-henrico.opendata.arcgis.com/). Census ACS tract-level income and demographics (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html).

Expected failure modes. Sale price is not market value — foreclosures, family transfers, and teardowns pollute the ratio. Assessment year and sale date must be aligned or the ratio is meaningless. Small tracts produce wild ratios from three sales. The standard statistics here (price-related differential, coefficient of dispersion) have established definitions the student should use rather than invent.

Done looks like. A sales-ratio study using recognized measures, with arm's-length filtering documented, tract-level results mapped, and a clear statement of where the sample was too thin to conclude anything.

Five C's. CT: distinguishing a real pattern from small-sample noise. CR: building a defensible arm's-length filter from messy transaction records. CO: having a peer re-run the filter independently and comparing counts. CM: a two-page finding a non-technical reader could bring to a public meeting. CZ: this is a tax-fairness question about real neighbors — the standard of evidence is correspondingly high.

Mentor role (advisory). An assessor, appraiser, real-estate analyst, or municipal finance staffer reviews the filtering logic at Checkpoint 1 and the conclusions at Checkpoint 2. School-supervised.

Rubric calibration. R1: asks a ratio question, not a "are assessments accurate" question. R2: every exclusion rule is written down and counted. R3: uses standard ratio statistics correctly. R4: reports confidence, and refuses thin tracts. R5: readable by a homeowner. R6: states plainly what the study cannot establish about intent.

Two ways this goes wrong. (a) The student divides assessment by sale price for every record, finds "regressivity," and has actually found foreclosures. (b) The student concludes the county is discriminating, on 40 sales, with no uncertainty interval — an overclaim that would not survive one question from a county analyst.

DS-03 · Tomorrow's air, from today's air and weather

The question. Can you predict tomorrow's peak PM2.5 at a Richmond-area monitor better than "same as today"? Persistence is a genuinely hard baseline in air quality.

Data. EPA Air Quality System pre-generated daily files (UNVERIFIED — https://aqs.epa.gov/aqsweb/airdata/download_files.html). NOAA GHCN-Daily weather (UNVERIFIED — https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily).

Expected failure modes. Monitors go offline; missing days imputed as zero produce clean air that never happened. Wildfire smoke events are massive outliers that dominate error and are not predictable from local weather. Using tomorrow's weather as a feature is fine if forecast data is used, but using tomorrow's observed weather is leakage.

Done looks like. A next-day forecast evaluated against persistence and against a seasonal-average baseline, with the smoke-event days identified and reported separately rather than deleted quietly.

Five C's. CT: recognizing that beating persistence is the real bar. CR: engineering features from wind and temperature that encode stagnation. CO: splitting monitor sites across a pair and comparing generalization. CM: a forecast card an asthma patient could act on. CZ: who changes their behavior on a bad-air day, and what a false "clean" prediction costs them.

Mentor role (advisory). An air-quality scientist, environmental engineer, or meteorologist reviews feature design at Checkpoint 1 and leakage checks at Checkpoint 2. School-supervised.

Rubric calibration. R1: one pollutant, one horizon, one site family. R2: missingness handled explicitly, never zero-filled. R3: persistence baseline present and beaten, or honestly not beaten. R4: outlier days reported, not deleted. R5: an actionable card. R6: false-negative cost stated in health terms.

Two ways this goes wrong. (a) Missing days become zeros, the model learns that missing means clean, and the metric looks superb. (b) The student deletes wildfire days as "outliers," reports strong performance, and has built a model that fails precisely on the days anyone needed it.

DS-04 · What actually predicts chronic absenteeism?

(This is the worked sample. If you read one brief before deciding whether this bank is any good, read this one.)

The question. Virginia publishes chronic absenteeism rates for every school, alongside enrollment, demographics, per-pupil spending, and outcome measures. Which school-level factors actually associate with chronic absenteeism — and which of the ones people assume matter do not survive controls?

Data. VDOE School Quality Profiles, downloadable build-a-table exports (UNVERIFIED — https://schoolquality.virginia.gov/). Census ACS tract or ZIP-level income and vehicle access, joined on school location (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html). Optionally VDOE fall membership and finance files (UNVERIFIED — https://www.doe.virginia.gov/data-policy-funding/data-reports).

Expected failure modes. This is school-level data, so every finding is ecological — a correlation across schools says nothing reliable about individual students, and students consistently write conclusions the unit of analysis does not support. Suppressed cells for small subgroups are not missing at random; dropping them removes exactly the schools most worth studying. Per-pupil spending is partly caused by need, so it will look like spending increases absenteeism. Post-2020 years are structurally different and should not be pooled with earlier ones without saying so.

Done looks like. A regression or tree model over school-level features predicting chronic absenteeism rate, with a stated baseline (state mean, or division mean), a clear statement of the ecological limitation, sensitivity analysis showing which findings survive adding division as a control, and one named variable the student expected to matter that did not.

Five C's. CT: separating association from cause, and saying which is which in writing. CR: constructing a join between school location and neighborhood data that does not exist in either source. CO: a structured peer critique where a partner attacks the causal language specifically. CM: a two-page brief written for a school board member, not a statistician. CZ: absenteeism is used to allocate resources and to blame families — naming that tension is part of the work.

Mentor role (advisory). A district data analyst, education researcher, school counselor, or public-policy analyst reviews the modeling plan at Checkpoint 1 and reads the draft brief at Checkpoint 2, with a specific instruction: challenge every causal verb. School-supervised, both sessions, never one-to-one unsupervised.

Rubric calibration. R1: the question is answerable at school level and the student says so. R2: suppressed cells counted and their effect on the sample described. R3: a real baseline, and a model whose gain over it is explained. R4: sensitivity to controls shown, not just one model. R5: a board member could read it and not be misled. R6: the student explicitly declines at least one conclusion the data cannot support.

Two ways this goes wrong. (a) The student finds that higher per-pupil spending predicts higher absenteeism and writes that spending does not help — having discovered reverse causation and reported it as policy. (b) The student drops every suppressed row for cleanliness, silently removing small and high-poverty schools, and reports a relationship that exists only in the schools large enough to be published.

DS-05 · Forecasting the James

The question. Given upstream gauge readings and rainfall, can you predict James River stage at Richmond 12 hours ahead well enough to be useful to someone deciding whether to move a boat?

Data. USGS National Water Information System instantaneous values (UNVERIFIED — https://waterdata.usgs.gov/nwis). NOAA precipitation (UNVERIFIED — https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily).

Expected failure modes. River stage is highly autocorrelated — a naive "same as now" model is very strong at 12 hours and students routinely fail to beat it. Flood events are rare and are the only ones that matter; a model trained on all data optimizes for tranquility. Upstream travel time is a real physical lag that must be discovered, not assumed.

Done looks like. A 12-hour-ahead stage forecast evaluated separately on ordinary conditions and on the highest-decile events, against persistence, with the discovered upstream lag reported.

Five C's. CT: understanding why persistence is hard to beat. CR: finding the lag empirically via cross-correlation. CO: dividing gauge sites and merging findings. CM: a plain-language risk statement. CZ: flood warnings are life-safety — overconfidence has a cost.

Mentor role (advisory). A hydrologist, civil engineer, or emergency-management analyst reviews lag analysis at Checkpoint 1 and event-conditional evaluation at Checkpoint 2. School-supervised.

Rubric calibration. R1: horizon and site fixed and justified. R2: gauge outages handled. R3: persistence beaten, or the failure honestly reported. R4: rare-event performance reported separately — this is the whole assignment. R5: a boater could use it. R6: states clearly it is not a flood warning system.

Two ways this goes wrong. (a) The student reports R² of 0.99, which persistence also achieves, and concludes the model works. (b) The student trains on all hours, never evaluates on flood events, and ships a model that is worst exactly when stage matters.

DS-06 · Where a delay at RIC actually comes from

The question. Airline delays propagate: a late inbound aircraft makes the next departure late. Using tail numbers, can you quantify how much of a departure delay at Richmond International is inherited rather than locally caused?

Data. BTS Marketing Carrier On-Time Performance (UNVERIFIED — https://www.transtats.bts.gov/ONTIME/). NOAA weather for the same days (UNVERIFIED — https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily).

Expected failure modes. Cancelled flights are not delays and must not be treated as very large ones. Tail-number chaining requires careful same-day ordering and breaks across midnight. Delay cause codes are self-reported by carriers and are not ground truth. RIC is a small airport — sample sizes per carrier per month get thin fast.

Done looks like. A quantified decomposition of departure delay into inherited versus originated, with the chaining logic documented and validated on a sample by hand, and a statement of which carriers had enough flights to support a conclusion.

Five C's. CT: recognizing the self-reported cause codes are a claim, not a fact. CR: reconstructing aircraft rotations from flat records. CO: hand-validating a peer's chains. CM: an explanation a frequent flyer would find genuinely useful. CZ: who bears the cost of delay, and how compensation rules treat "weather" versus "carrier."

Mentor role (advisory). An operations analyst, airline or airport employee, or logistics professional reviews chaining at Checkpoint 1 and the decomposition at Checkpoint 2. School-supervised.

Rubric calibration. R1: scoped to RIC and a defined period. R2: cancellations handled explicitly. R3: a baseline attributing all delay locally. R4: hand-validation of the chain logic on real rows. R5: useful to a traveler. R6: treats carrier-reported causes skeptically.

Two ways this goes wrong. (a) Cancellations are coded as 999-minute delays and one storm week dominates every result. (b) The student trusts the cause codes, reports that weather causes most delay, and has reported what carriers chose to file.

DS-07 · Who can actually reach a grocery store?

The question. Food access is usually measured as straight-line distance to a supermarket. Using road networks and transit, how different does access look when measured by travel time — and for whom does the answer change most?

Data. USDA Food Access Research Atlas (UNVERIFIED — https://www.ers.usda.gov/data-products/food-access-research-atlas/). OpenStreetMap road network and retail POIs (UNVERIFIED — https://www.openstreetmap.org/). Census ACS vehicle availability and income (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html). GRTC GTFS for transit travel time (UNVERIFIED — https://ridegrtc.com/developer-resources/).

Expected failure modes. OSM retail tagging is inconsistent — dollar stores, corner stores, and full supermarkets are not reliably distinguished, and that distinction is the entire analysis. Straight-line and network distance diverge most exactly where rivers and highways are, which is where it matters. Transit travel time depends on time of day.

Done looks like. A comparison of straight-line, driving, and transit-based access for Richmond-area tracts, with the store classification rules written down, and a named set of tracts whose classification flips depending on method.

Five C's. CT: recognizing the measure defines the finding. CR: building a store classifier from inconsistent tags. CO: independent double-coding of a store sample to measure agreement. CM: a map a planner could use. CZ: "food desert" designations affect funding — the definition is a political object.

Mentor role (advisory). An urban planner, GIS analyst, or public-health professional reviews classification rules at Checkpoint 1 and the map at Checkpoint 2. School-supervised.

Rubric calibration. R1: compares methods rather than asserting one. R2: store coding rules documented and inter-rater agreement reported. R3: straight-line as the baseline method. R4: identifies where methods disagree, and why. R5: map is legible and honest about uncertainty. R6: addresses how the label affects real funding.

Two ways this goes wrong. (a) Every OSM node tagged shop=convenience counts as a grocery store and half the food deserts disappear. (b) The student computes transit time at 9am Tuesday only, and reports access that does not exist for a night-shift worker.

DS-08 · Broadband, income, and what the map does not show

The question. FCC broadband availability data is self-reported by providers at a granular level. Does reported availability match what the ACS says about actual household subscription — and where is the gap largest?

Data. FCC Broadband Data Collection / National Broadband Map (UNVERIFIED — https://broadbandmap.fcc.gov/data-download). Census ACS internet subscription and income tables (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html).

Expected failure modes. Availability and adoption are different things and conflating them is the classic error — a place can be fully served and barely subscribed, which is an affordability story, not an infrastructure one. FCC data is provider-reported and historically overstates coverage. Geographic units differ between sources and require careful crosswalking.

Done looks like. A tract-level comparison of reported availability against measured subscription, with income controlled, and an explicit separation of the "no infrastructure" story from the "cannot afford it" story.

Five C's. CT: holding two similar-sounding variables apart. CR: crosswalking incompatible geographies defensibly. CO: peer review of the crosswalk. CM: a brief that does not let a reader collapse availability into adoption. CZ: broadband funding is allocated on these maps.

Mentor role (advisory). A telecom engineer, digital-equity organizer, or policy analyst reviews the crosswalk at Checkpoint 1 and the framing at Checkpoint 2. School-supervised.

Rubric calibration. R1: the availability/adoption distinction is in the question itself. R2: crosswalk method documented with loss reported. R3: income-only baseline. R4: gap quantified with uncertainty. R5: a reader cannot conflate the two. R6: names the funding consequence of overstated coverage.

Two ways this goes wrong. (a) The student reports "20% of the county lacks broadband" using subscription data, and has measured poverty. (b) The student treats provider-reported availability as ground truth and concludes there is no access problem.

DS-09 · Crash severity, and the limits of prediction

The question. Given crash circumstances — road type, lighting, weather, restraint use — how well can severity be predicted, and what happens to the model's usefulness when you remove the variables a policymaker cannot change?

Data. NHTSA Fatality Analysis Reporting System (UNVERIFIED — https://www.nhtsa.gov/file-downloads?p=nhtsa/downloads/FARS/). Optionally Virginia DOT traffic volume counts for exposure (UNVERIFIED — https://www.virginiadot.org/info/ct-trafficcounts.asp).

Content note. FARS records fatal crashes. It is real people. Set expectations before assigning, and let a student choose a different brief without explanation.

Expected failure modes. FARS contains only fatal crashes, so "predicting severity" inside it is a truncated problem and students routinely forget this. Without exposure data, counts measure traffic volume, not risk. Restraint use is strongly predictive and completely uninteresting as a finding — the assignment is what remains after it.

Done looks like. A severity or outcome model built on FARS with the selection bias stated up front, a version excluding non-actionable variables, and a short list of road-design factors that survive.

Five C's. CT: reasoning about a truncated sample. CR: constructing an exposure denominator from a separate source. CO: peer challenge on the selection-bias framing. CM: writing about fatality data without sensationalizing. CZ: recommendations here affect road funding and the people who use those roads.

Mentor role (advisory). A traffic engineer, transportation planner, or public-health epidemiologist reviews the exposure approach at Checkpoint 1 and the recommendations at Checkpoint 2. School-supervised.

Rubric calibration. R1: the truncation is acknowledged in the question. R2: exposure denominator sourced or its absence stated. R3: a baseline using only restraint and speed. R4: performance reported on actionable variables alone. R5: sober, non-sensational writing. R6: refuses to rank roads without exposure data.

Two ways this goes wrong. (a) The student reports the deadliest intersections and has found the busiest ones. (b) The student builds a 94%-accurate model that is mostly "was the person wearing a seatbelt" and presents it as a road-safety finding.

DS-10 · Forecasting the grid

The question. Can you forecast next-day hourly electricity demand for the PJM region better than a "same hour last week" baseline, and where does the model break?

Data. EIA Open Data hourly electric grid monitor (UNVERIFIED — https://www.eia.gov/opendata/). NOAA temperature (UNVERIFIED — https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily).

Expected failure modes. Demand is driven by temperature non-linearly and in both directions — heating below, cooling above, flat in between. Holidays behave like Sundays and wreck a naive weekly baseline. Daylight-saving transitions produce a duplicated and a missing hour that silently corrupt joins.

Done looks like. An hourly day-ahead forecast beating the seasonal-naive baseline, with a temperature response curve the student can show and explain, and holiday and DST handling documented.

Five C's. CT: recognizing a U-shaped relationship and not forcing a line through it. CR: encoding calendar effects. CO: splitting seasons across a pair and comparing. CM: a chart that makes the temperature response obvious. CZ: demand forecasting errors cause either blackouts or wasted generation.

Mentor role (advisory). A utility analyst, energy engineer, or data scientist reviews feature design at Checkpoint 1 and error analysis at Checkpoint 2. School-supervised.

Rubric calibration. R1: horizon and region fixed. R2: DST and holidays handled explicitly. R3: seasonal-naive baseline present. R4: worst errors located in time and explained. R5: the temperature curve is legible. R6: notes the asymmetric cost of under-forecasting.

Two ways this goes wrong. (a) A linear temperature term makes the model worse in both summer and winter and the student concludes weather does not matter. (b) The DST duplicate hour shifts a whole year's join by one hour and the model looks mysteriously bad.

DS-11 · The hot side of the street

The question. Urban heat islands track tree canopy and impervious surface. Within Richmond, how much of surface temperature variation is explained by canopy — and does the pattern line up with historical redlining boundaries?

Data. NASA POWER or Landsat-derived surface temperature (UNVERIFIED — https://power.larc.nasa.gov/). Richmond open data tree canopy and land cover (UNVERIFIED — https://data.richmondgov.com/). Mapping Inequality HOLC redlining polygons (UNVERIFIED — https://dsl.richmond.edu/panorama/redlining/). Census ACS for current demographics (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html).

Expected failure modes. Satellite surface temperature is not air temperature and the two are routinely confused. Resolution mismatch between temperature rasters and canopy polygons requires an explicit aggregation choice. HOLC boundaries are from the 1930s and present-day demographics have shifted — the historical claim and the current claim are different claims.

Done looks like. A model of surface temperature from canopy and impervious cover, with residuals mapped, and a separate, carefully worded comparison against HOLC grades that does not overstate causality across ninety years.

Five C's. CT: separating a historical association from a present-day mechanism. CR: resolving raster-to-polygon aggregation. CO: peer check on the spatial join. CM: a map essay. CZ: this is the most direct environmental-justice question in the bank and demands precision, not indignation.

Mentor role (advisory). A GIS analyst, urban forester, climate scientist, or environmental-justice researcher reviews the spatial method at Checkpoint 1 and the historical framing at Checkpoint 2. School-supervised.

Rubric calibration. R1: surface vs. air temperature distinguished in the question. R2: aggregation choice stated and defended. R3: canopy-only baseline. R4: residuals mapped, not just summarized. R5: the map essay stands alone. R6: the ninety-year causal gap is named rather than implied.

Two ways this goes wrong. (a) The student writes "redlining causes heat" from a correlation across historical districts, with no mechanism and no controls. (b) Surface temperature is reported as what a person feels, overstating the effect by several degrees.

DS-12 · A housing model that knows what it does not know

The question. Build a home-value model for a Richmond-area ZIP set. Then do the part that matters: find the segments where it is systematically wrong, and decide whether it should be used at all.

Data. Zillow Home Value Index research data (UNVERIFIED — https://www.zillow.com/research/data/). Census ACS structure age, tenure, income (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html). Local assessment records (UNVERIFIED — https://data-henrico.opendata.arcgis.com/).

Expected failure modes. ZHVI is already a smoothed index, not raw transactions — modeling it is modeling someone else's model. Aggregate-level fit hides neighborhood-level failure. Automated valuation models have a documented history of underperforming in majority-Black neighborhoods; a student who does not check for this has skipped the assignment.

Done looks like. A value model with error decomposed by neighborhood characteristics, an explicit fairness check across neighborhood demographics, and a written recommendation about deployment — including, if warranted, "do not deploy."

Five C's. CT: understanding what modeling an index means. CR: designing a segment-level error audit. CO: adversarial peer review looking for hidden failure. CM: a model card with an intended-use and out-of-scope section. CZ: valuation models affect wealth; a recommendation not to deploy is a legitimate result.

Mentor role (advisory). An appraiser, housing-policy researcher, or fair-lending analyst reviews the audit design at Checkpoint 1 and the deployment recommendation at Checkpoint 2. School-supervised.

Rubric calibration. R1: the target is understood as an index. R2: sources and vintages aligned. R3: a simple baseline (median by ZIP). R4: error decomposed by segment, not just overall. R5: a real model card. R6: an explicit deployment stance with reasons.

Two ways this goes wrong. (a) The student reports 8% median error and stops, never finding the 30% error in two neighborhoods. (b) The student runs a fairness check, finds disparity, notes it in one sentence, and recommends deployment anyway.

DS-13 · Are the birds moving north?

The question. Pick a species with dense occurrence records. Has the centroid or northern edge of its observed range shifted over the last thirty years — and how much of any apparent shift is just more people with phones?

Data. GBIF species occurrence downloads (UNVERIFIED — https://www.gbif.org/). NOAA climate normals (UNVERIFIED — https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily).

Expected failure modes. Observation effort has exploded since smartphone apps arrived; raw occurrence counts measure observers, not organisms. Records cluster along roads and near cities. Taxonomic revisions split and merge species across the study period.

Done looks like. A range-shift estimate with an explicit effort correction, the correction method defended, and a sensitivity analysis showing how the conclusion changes under different corrections.

Five C's. CT: distinguishing signal from sampling. CR: constructing an effort proxy. CO: comparing species with a partner to see whether patterns are shared. CM: a figure that shows the correction's effect. CZ: climate evidence gets scrutinized adversarially — the work must survive it.

Mentor role (advisory). An ecologist, ornithologist, museum collections manager, or biostatistician reviews the effort correction at Checkpoint 1 and the sensitivity analysis at Checkpoint 2. School-supervised.

Rubric calibration. R1: one species, one measure of range. R2: taxonomic changes checked. R3: uncorrected trend as the baseline. R4: sensitivity across correction choices. R5: the corrected-vs-raw figure is the centerpiece. R6: states what confounders remain.

Two ways this goes wrong. (a) The student plots raw occurrences over time and discovers iNaturalist's launch date. (b) One effort correction is applied, produces the expected answer, and no alternative is tested.

DS-14 · A recommender that is not just popular

The question. Build a movie recommender. Then measure what it does to the tail: does it surface anything a user would not have found anyway, and what happens to a brand-new user or a brand-new item?

Data. MovieLens datasets (UNVERIFIED — https://grouplens.org/datasets/movielens/).

Expected failure modes. Recommending the most popular items scores deceptively well on accuracy metrics — popularity is a very strong baseline and beating it requires measuring something else. Random train/test splits leak the future into the past; a time-based split is required. Cold start is the actual hard problem and is easy to avoid testing.

Done looks like. A recommender evaluated on ranking metrics and on coverage and novelty, with a time-based split, and an explicit cold-start experiment for new users and new items.

Five C's. CT: recognizing that accuracy and usefulness diverge here. CR: designing a novelty metric that means something. CO: a peer blind-rates recommendation lists for quality. CM: explaining the popularity-versus-discovery tradeoff to a non-specialist. CZ: filter bubbles and homogenization are the citizenship question.

Mentor role (advisory). A recommender-systems or ML engineer, product analyst, or librarian reviews the metric set at Checkpoint 1 and the cold-start results at Checkpoint 2. School-supervised.

Rubric calibration. R1: the question is about tail behavior, not just accuracy. R2: time-based split implemented correctly. R3: popularity baseline, honestly reported. R4: coverage and novelty measured alongside ranking. R5: the tradeoff is explained clearly. R6: addresses homogenization.

Two ways this goes wrong. (a) A random split lets the model see future ratings; metrics are excellent and meaningless. (b) The model beats popularity by 2% on one ranking metric while recommending the same 50 films to everyone, and coverage is never measured.

DS-15 · Messy labels, real products

The question. Open Food Facts is crowd-sourced, multilingual, and inconsistent. Can you predict a product's nutrition grade from its ingredient text — and what does the failure pattern tell you about who contributes data?

Data. Open Food Facts full database export (UNVERIFIED — https://world.openfoodfacts.org/data).

Expected failure modes. Nutrition grade is partly computed from nutrition fields already in the dataset — including those as features is leakage and produces a perfect, useless model. Coverage is far denser for French and European products, so a model trained on everything is a model of European groceries. Ingredient text is multilingual and inconsistently delimited.

Done looks like. A text-based classifier with the computed-field leakage explicitly excluded and that exclusion documented, performance reported per language or region, and a short analysis of where coverage is thin.

Five C's. CT: finding the leakage before the metric flatters you. CR: text normalization across languages. CO: peer audit specifically hunting for leaked features. CM: a model card naming the geographic bias. CZ: whose food is well-represented in open data, and whose is not.

Mentor role (advisory). A data scientist, nutritionist, or food-industry analyst reviews the feature audit at Checkpoint 1 and the bias analysis at Checkpoint 2. School-supervised.

Rubric calibration. R1: prediction from text specifically. R2: leakage audit performed and written up. R3: a majority-class baseline. R4: performance disaggregated by region. R5: model card states the population it works for. R6: coverage inequity named.

Two ways this goes wrong. (a) Sugar and fat per 100g stay in the feature set, the model hits 99%, and it has re-derived the formula. (b) Aggregate accuracy is reported for a model that works on French products and fails on everything else.

DS-16 · What the coverage emphasizes

The question. Take one well-defined local or regional news topic over a fixed window. Do outlets differ measurably in what they emphasize — which entities and themes co-occur — and can you show it without claiming to have measured bias?

Data. GDELT Global Knowledge Graph (UNVERIFIED — https://www.gdeltproject.org/data.html).

Expected failure modes. GDELT's tone and theme codes are automated and noisy; treating them as ground truth is the central error. Outlet volume varies by orders of magnitude, so raw counts measure output, not emphasis. "Bias" is not measurable with this data, and students reach for the word immediately.

Done looks like. A comparative emphasis analysis — normalized entity and theme co-occurrence by outlet — with automated-coding limitations stated, and a deliberate, written refusal to characterize outlets as biased.

Five C's. CT: distinguishing emphasis from bias rigorously. CR: designing a volume-normalized comparison. CO: peer replication on a different topic window. CM: a finding that survives a hostile reader. CZ: media criticism is high-stakes; precision is the ethic.

Mentor role (advisory). A journalist, media researcher, computational social scientist, or librarian reviews the framing at Checkpoint 1 and the language of the conclusions at Checkpoint 2. School-supervised.

Rubric calibration. R1: scoped to emphasis, one topic, one window. R2: GDELT coding limits stated. R3: raw-count comparison as the naive baseline. R4: normalization effect shown. R5: language is precise throughout. R6: the refusal to claim bias is explicit and reasoned.

Two ways this goes wrong. (a) The student averages GDELT tone by outlet and publishes a bias ranking built on automated sentiment. (b) Raw article counts are compared and the finding is that large outlets publish more.

DS-17 · What a job posting actually asks for

The question. Using occupational skill data, map how skill requirements differ across occupations in one Virginia industry — and identify which skills are shared across occupations that people think of as unrelated.

Data. O*NET database (UNVERIFIED — https://www.onetcenter.org/database.html). BLS Occupational Employment and Wage Statistics for Virginia (UNVERIFIED — https://www.bls.gov/oes/tables.htm).

Expected failure modes. O*NET importance ratings are survey-derived and compress into a narrow band, so raw distances between occupations are dominated by scale rather than content. Occupation codes change across releases. Correlating skills with wages invites immediate causal overreach.

Done looks like. A clustering or similarity analysis over occupations with the standardization choice defended, at least one non-obvious adjacency identified and sanity-checked, and wage discussion kept explicitly descriptive.

Five C's. CT: recognizing scale effects in similarity. CR: finding a genuinely surprising adjacency. CO: peer sanity-check of clusters against lived knowledge. CM: a career-counseling-usable summary. CZ: this maps onto real students' futures — overclaiming has a cost.

Mentor role (advisory). A workforce-development analyst, HR professional, career counselor, or economist reviews the similarity method at Checkpoint 1 and the adjacencies at Checkpoint 2. School-supervised.

Rubric calibration. R1: one industry, one similarity question. R2: code vintages aligned. R3: a naive similarity baseline. R4: clusters validated against outside knowledge. R5: usable by a counselor. R6: no causal wage claims.

Two ways this goes wrong. (a) Unstandardized ratings produce clusters that reflect how generous each survey panel was. (b) The student reports that a skill "raises wages" from a cross-sectional correlation.

DS-18 · Predicting the health inspection

The question. Using restaurant inspection history, can you predict which establishments will fail their next inspection — and should a health department actually use such a model to target visits?

Data. Virginia restaurant inspection records (UNVERIFIED — https://www.vdh.virginia.gov/environmental-health/food-safety-in-virginia/) or the Richmond open data portal (UNVERIFIED — https://data.richmondgov.com/).

Expected failure modes. Inspection frequency is itself risk-based, so past inspection count is a proxy for past suspicion — a feedback loop. Establishments that closed vanish from the data. Violation severity coding changes over time.

Done looks like. A next-inspection risk model with the feedback loop explicitly analyzed, a discussion of what targeted enforcement would do to small and independently-owned establishments, and a deployment recommendation.

Five C's. CT: identifying the feedback loop unprompted. CR: constructing a survivorship-corrected sample. CO: peer challenge on deployment ethics. CM: a recommendation memo to a health department. CZ: enforcement targeting has distributional consequences.

Mentor role (advisory). An environmental-health specialist, restaurant operator, or public-sector data analyst reviews the loop analysis at Checkpoint 1 and the memo at Checkpoint 2. School-supervised.

Rubric calibration. R1: framed as a targeting decision, not just prediction. R2: closures and coding changes handled. R3: a "inspect everyone equally" baseline. R4: performance plus a feedback-loop simulation or argument. R5: a real memo. R6: distributional impact addressed with a stance.

Two ways this goes wrong. (a) Prior inspection count is the top feature and the model has learned who was already suspected. (b) The student recommends targeting without noticing it concentrates on small independent restaurants.

DS-19 · Fifteen minutes to a park

The question. What fraction of Henrico residents live within a 15-minute walk of a public park or library, and how does that fraction differ across income and age groups?

Data. Henrico open GIS parks, libraries, and boundaries (UNVERIFIED — https://data-henrico.opendata.arcgis.com/). OpenStreetMap pedestrian network (UNVERIFIED — https://www.openstreetmap.org/). Census ACS demographics (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html).

Expected failure modes. Straight-line buffers ignore highways, rivers, and missing sidewalks — the classic "park across the interstate" error. OSM sidewalk data is incomplete in suburban areas, so a walkable network may be modeled where none exists. Park polygons include inaccessible land.

Done looks like. A network-based walkshed analysis with barriers respected, sidewalk data completeness assessed and reported, and access rates broken out by demographic group.

Five C's. CT: understanding what a buffer assumes. CR: handling incomplete sidewalk data honestly. CO: ground-truthing a sample of walksheds with a partner. CM: a map for a parks department. CZ: park access is a well-documented equity issue.

Mentor role (advisory). A parks planner, GIS analyst, or public-health professional reviews the network method at Checkpoint 1 and the equity analysis at Checkpoint 2. School-supervised.

Rubric calibration. R1: walkshed, not buffer. R2: sidewalk completeness quantified. R3: straight-line buffer as baseline. R4: difference between methods reported. R5: a usable map. R6: names where data gaps could mask inequity.

Two ways this goes wrong. (a) A half-mile circular buffer counts households separated from the park by I-64. (b) Missing sidewalks are treated as walkable and suburban access is overstated across the board.

DS-20 · Small-area population forecasting

The question. Forecast tract-level population five years out for a Richmond-area county. Then do the real work: quantify how wrong such forecasts typically are, by backtesting against tracts you already know the answer for.

Data. Census ACS 5-year time series (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html). Census building permits (UNVERIFIED — https://www.census.gov/construction/bps/).

Expected failure modes. ACS 5-year estimates are overlapping rolling averages — treating consecutive vintages as independent annual observations badly understates volatility. Tract boundaries change between decennial censuses. Margins of error on small tracts are large and usually ignored.

Done looks like. A backtested forecast with prediction intervals that account for ACS margins of error, boundary changes handled explicitly, and a plain statement of the tract sizes at which forecasting stops being meaningful.

Five C's. CT: understanding overlapping-sample structure. CR: designing a backtest with limited history. CO: comparing interval calibration with a peer. CM: communicating uncertainty to a planner. CZ: population forecasts drive school siting and service allocation.

Mentor role (advisory). A demographer, regional planner, or statistician reviews the backtest design at Checkpoint 1 and interval calibration at Checkpoint 2. School-supervised.

Rubric calibration. R1: forecast horizon and geography fixed. R2: boundary changes handled. R3: a naive persistence or linear-trend baseline. R4: intervals are calibrated, not decorative. R5: a planner could act on it. R6: states the minimum tract size for a credible forecast.

Two ways this goes wrong. (a) Overlapping ACS vintages are treated as independent years and the intervals come out absurdly tight. (b) A point forecast is delivered with no interval, for tracts whose margin of error exceeds the predicted change.

DS-21 · What a storm costs

The question. Using historical storm event records, model reported property damage from event characteristics — and then confront how badly damage reporting behaves as a target variable.

Data. NOAA Storm Events Database (UNVERIFIED — https://www.ncdc.noaa.gov/stormevents/). Census ACS housing values for exposure (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html).

Expected failure modes. Damage figures are entered inconsistently, sometimes as strings like "2.5M", and are not inflation-adjusted. Reporting completeness has improved over decades, so trends conflate real change with better record-keeping. Zero-damage events vastly outnumber costly ones, producing a heavily skewed target.

Done looks like. A damage model with inflation adjustment applied, the skewed target handled deliberately (transformation or a two-stage approach), and reporting-completeness change addressed before any trend claim.

Five C's. CT: separating reporting change from real change. CR: handling a zero-inflated skewed target. CO: peer audit of the parsing logic. CM: a risk summary for an emergency manager. CZ: disaster funding decisions rest on these records.

Mentor role (advisory). An emergency manager, insurance analyst, or climate risk professional reviews the target handling at Checkpoint 1 and trend claims at Checkpoint 2. School-supervised.

Rubric calibration. R1: scoped to event type and region. R2: damage strings parsed and inflation-adjusted, with the parser validated. R3: a mean-damage baseline. R4: performance reported on the costly tail specifically. R5: usable by an emergency manager. R6: no trend claim without a completeness argument.

Two ways this goes wrong. (a) "2.5M" parses as 2.5 and a tornado outbreak becomes cheaper than a hailstorm. (b) The student reports rising storm damage and has measured improved reporting plus inflation.

DS-22 · Predicting graduation, and deciding not to

The question. Using institution-level higher-education data, model graduation rates from institutional characteristics. Then answer the harder question: what would go wrong if an admissions office used this model on individuals?

Data. IPEDS institutional data (UNVERIFIED — https://nces.ed.gov/ipeds/use-the-data). College Scorecard (UNVERIFIED — https://collegescorecard.ed.gov/data/).

Expected failure modes. Institution-level patterns say nothing valid about individual students — the ecological fallacy again, and here it has direct human consequences. Selectivity and graduation rate are near-tautologically linked. Small institutions produce extreme rates from tiny cohorts.

Done looks like. An institution-level model with the tautology addressed, plus a written analysis of why applying it to individuals is invalid — and what a school would have to do differently to make an individual-level prediction defensible.

Five C's. CT: the ecological fallacy, stated precisely. CR: separating selectivity from institutional effect. CO: peer debate on the deployment question. CM: a memo a counselor could use. CZ: predictive models in education already steer real students.

Mentor role (advisory). A higher-education researcher, admissions professional, or school counselor reviews the model at Checkpoint 1 and the deployment argument at Checkpoint 2. School-supervised.

Rubric calibration. R1: institution-level framing is explicit. R2: small-cohort institutions handled. R3: a selectivity-only baseline — and the student notices how strong it is. R4: residual analysis identifying over- and under-performing institutions. R5: a counselor-usable memo. R6: a clear, reasoned argument against individual-level use.

Two ways this goes wrong. (a) The model is 90% selectivity and is presented as measuring institutional quality. (b) The student suggests the model could flag at-risk applicants — exactly the invalid inference the brief is built to prevent.

DS-23 · Is the water safe, and how would you know?

The question. Using water-quality monitoring records for a Virginia watershed, classify whether a sample exceeds a standard — and quantify how much monitoring gaps limit what can be said.

Data. USGS Water Quality Portal (UNVERIFIED — https://www.waterqualitydata.us/). EPA water quality standards documentation (UNVERIFIED — https://www.epa.gov/waterdata/water-quality-data). USGS NWIS streamflow (UNVERIFIED — https://waterdata.usgs.gov/nwis).

Expected failure modes. Detection limits mean many readings are "below X" rather than numbers; substituting zero or the limit itself biases results in opposite directions. Sampling is not random — agencies sample when they suspect problems, and often after rain. Different labs and methods are not directly comparable.

Done looks like. A classification model with non-detects handled by a defensible, named method, sampling bias discussed, and results reported per parameter rather than pooled into a single "safe/unsafe" verdict.

Five C's. CT: recognizing censored data as a distinct statistical problem. CR: choosing and defending a non-detect method. CO: peer replication under a different method. CM: a public-facing summary that does not alarm or reassure beyond the evidence. CZ: drinking and recreational water safety is a direct public-health claim.

Mentor role (advisory). An environmental scientist, water utility analyst, or watershed-group staffer reviews non-detect handling at Checkpoint 1 and public framing at Checkpoint 2. School-supervised.

Rubric calibration. R1: one watershed, named parameters. R2: non-detects handled with a named method and sensitivity shown. R3: a base-rate baseline. R4: per-parameter results. R5: calibrated public language. R6: explicitly declines to declare water "safe."

Two ways this goes wrong. (a) Non-detects become zeros and average concentrations drop below reality. (b) The student pools parameters into one verdict and publishes that a creek is unsafe on the strength of one exceedance.

DS-24 · How many cars, really?

The question. VDOT publishes traffic counts at limited locations. Can you estimate volume on unmonitored segments from road characteristics and network position — and how confident should anyone be in those estimates?

Data. VDOT traffic volume counts (UNVERIFIED — https://www.virginiadot.org/info/ct-trafficcounts.asp). OpenStreetMap road network (UNVERIFIED — https://www.openstreetmap.org/). Census ACS commuting data (UNVERIFIED — https://www.census.gov/data/developers/data-sets/acs-5year.html).

Expected failure modes. Counters are placed on important roads, so the training sample is not representative of the segments needing prediction — the extrapolation is the whole difficulty. Annual average daily traffic hides enormous variation. Spatial autocorrelation makes random cross-validation wildly optimistic; spatial cross-validation is required.

Done looks like. A volume model validated with spatially blocked cross-validation, with the difference from random CV reported explicitly, and a stated road class beyond which the model should not be trusted.

Five C's. CT: understanding why random CV lies here. CR: constructing network-position features. CO: peer implementation of the spatial split. CM: an uncertainty-carrying map for a planner. CZ: traffic estimates drive road spending and neighborhood impacts.

Mentor role (advisory). A transportation planner, traffic engineer, or GIS analyst reviews the validation design at Checkpoint 1 and the uncertainty map at Checkpoint 2. School-supervised.

Rubric calibration. R1: prediction on unmonitored segments is the stated goal. R2: counter placement bias documented. R3: a road-class-mean baseline. R4: spatial CV and random CV both reported, with the gap explained. R5: uncertainty visible on the map. R6: states where the model must not be used.

Two ways this goes wrong. (a) Random cross-validation puts adjacent segments in train and test; R² of 0.95 collapses on genuinely new roads. (b) Estimates are extended to residential streets from a model trained entirely on arterials.

DS-25 · Break a published result

The question. Choose a published model or benchmark result on a classic public dataset. Reproduce it. Then find the conditions under which it fails, and write the model card its authors did not.

Data. UCI Machine Learning Repository (UNVERIFIED — https://archive.ics.uci.edu/). Optionally a Hugging Face dataset with a published baseline (UNVERIFIED — https://huggingface.co/datasets). The chosen paper or model card must be publicly readable without a paywall.

Expected failure modes. Reproduction usually fails first for boring reasons — preprocessing, split seeds, library versions — and students conclude the result was wrong. Several beloved UCI datasets have known documented problems, which is a feature here. "Finding failure" can slide into cherry-picking a hostile subset.

Done looks like. A documented reproduction with any gap from the published number explained, a systematic robustness probe (distribution shift, subgroup performance, label noise, or feature ablation), and a model card with intended use, out-of-scope use, and measured limitations.

Five C's. CT: distinguishing a reproduction failure from a genuine refutation. CR: designing probes the original authors did not run. CO: swapping papers with a peer and attempting each other's reproduction. CM: writing a model card to a professional standard. CZ: reproducibility is a scientific-integrity practice, and this is where a student practices it.

Mentor role (advisory). An ML practitioner, research scientist, or academic reviews the reproduction plan at Checkpoint 1 and critiques the model card at Checkpoint 2. School-supervised.

Rubric calibration. R1: a specific, clearly-scoped published claim. R2: environment and preprocessing documented well enough for a third party. R3: the published number is the baseline. R4: robustness probes are systematic, not cherry-picked. R5: a professional-grade model card. R6: distinguishes "I could not reproduce this" from "this is wrong."

Two ways this goes wrong. (a) The student gets 4% below the published number, blames the authors, and never checks the preprocessing. (b) The student searches subgroups until one looks bad and reports it as a fundamental flaw, with no correction for multiple comparisons.

Checkpoints

Two, for every brief. They are the difference between a capstone and a panic.

Checkpoint 1 — week 3. Scope and data. The student demonstrates they can load the data and produce one honest summary of it. They state the baseline they intend to beat. Most capstone failures are prevented here, by cutting scope.

Checkpoint 2 — week 9. Error analysis. Not "is the model good" but "where is it wrong, and who does that hurt." The student brings failures, not just metrics.

The mentor, where one exists, joins both — in a school-supervised setting, never one-to-one.

What we do not claim

This bank does not produce verified credit. No AI or CS course in Virginia currently can — there is no CS SOL test and no CS or AI entry on the Board-approved substitute test list. For any currently enrolled student, a CTE credential does not convert to a verified credit either.
Mentorship is not HQWBL. Where these briefs touch work-based learning, it is through Internship or Entrepreneurship.
Every dataset URL here is UNVERIFIED. Check before you assign.

Found a source that has moved, or a brief that broke in the classroom? Tell us and we will fix it: hello@contact.schoolhouse.space

Applied ML Systems — free brief bank