Venture & Private Capital · Capstone brief
INV-05 · Build a screen, then try to break it
Build a scoring rubric that claims to pick better companies from public data. Backtest it, then try to break it: how much of its apparent skill is look-ahead or survivorship bias, and what is the honest confidence interval?
The question
Build a scoring rubric that claims to pick better companies from public data. Backtest it, then try to break it: how much of its apparent skill is look-ahead or survivorship bias, and what is the honest confidence interval?
Data room / materials
Public-company data the student can cite, such as companies that went public in a stated period, with figures from their public filings (https://www.sec.gov/edgar/search/). Lab L-11 first. A spreadsheet or Python. BUILD is stub-only: no live model, no API key. If any model is used to propose features, its suggestions are logged and tested like any other hypothesis.
Expected failure modes
Features that use information not available at the decision date. A universe built only from survivors. Tuning the rubric until the backtest looks good. Reporting a hit rate with no interval. Calling the result a stock-picking method.
Done looks like
A screen report: the universe and how it was built, including who is missing; the rubric, fixed before the backtest; the backtest result with a confidence interval (a Wilson interval is a good choice for a hit rate); a look-ahead audit of every feature; a survivorship estimate; and a conclusion that may be "the screen has no demonstrated skill."
Five C's
CT: distinguishing skill from leakage. CR: designing the attack on your own screen. CO: a peer gets the rubric without the results and tries to find the leak. CM: a report a statistician would accept. CZ: who is harmed when a biased screen is sold as a method.
Mentor role
A statistician, data scientist, or analyst reviews the look-ahead audit. Standing instruction: reject any backtest whose rubric changed after results were seen. School-supervised.
Rubric calibration
R1: one screen, one universe, one period. R2: data and code re-runnable. R3: comparator is a random or equal-weight pick. R4: look-ahead and survivorship quantified. R5: interval reported plainly. R6: refuses to call it a method if it has no demonstrated skill.
Two ways this goes wrong
(a) A brilliant backtest powered by a feature from the future. (b) An honest null result reported as a failure instead of a finding.
Credit lane fit
Lane A (statistics, computer science, or finance capstone). Pairs with Applied ML. No verified credit claim.
An investment study, not an investment. Fictional case files. Not legal, tax, or investment advice.