Applied ML Systems · Capstone brief
DS-14 · A recommender that is not just popular
Build a movie recommender. Then measure what it does to the tail: does it surface anything a user would not have found anyway, and what happens to a brand-new user or a brand-new item?
The question
Build a movie recommender. Then measure what it does to the tail: does it surface anything a user would not have found anyway, and what happens to a brand-new user or a brand-new item?
Data
MovieLens datasets (https://grouplens.org/datasets/movielens/).
Expected failure modes
Recommending the most popular items scores deceptively well on accuracy metrics — popularity is a very strong baseline and beating it requires measuring something else. Random train/test splits leak the future into the past; a time-based split is required. Cold start is the actual hard problem and is easy to avoid testing.
Done looks like
A recommender evaluated on ranking metrics and on coverage and novelty, with a time-based split, and an explicit cold-start experiment for new users and new items.
Five C's
CT: recognizing that accuracy and usefulness diverge here. CR: designing a novelty metric that means something. CO: a peer blind-rates recommendation lists for quality. CM: explaining the popularity-versus-discovery tradeoff to a non-specialist. CZ: filter bubbles and homogenization are the citizenship question.
Mentor role
A recommender-systems or ML engineer, product analyst, or librarian reviews the metric set at Checkpoint 1 and the cold-start results at Checkpoint 2. School-supervised.
Rubric calibration
R1: the question is about tail behavior, not just accuracy. R2: time-based split implemented correctly. R3: popularity baseline, honestly reported. R4: coverage and novelty measured alongside ranking. R5: the tradeoff is explained clearly. R6: addresses homogenization.
Two ways this goes wrong
(a) A random split lets the model see future ratings; metrics are excellent and meaningless. (b) The model beats popularity by 2% on one ranking metric while recommending the same 50 films to everyone, and coverage is never measured.
Source moved or something unclear? Send feedback on this brief. Mentors are advisory; the school supervises. These briefs do not produce verified computer-science credit.