Model-in-the-Loop Engineering · Capstone brief
ENG-08 · Calibration drift
After a teacher-defined distribution shift (seasonal temperature, wear, or load change), can you detect that the model’s errors drift — and say whether recalibration or human takeover is the right response?
The question
After a teacher-defined distribution shift (seasonal temperature, wear, or load change), can you detect that the model’s errors drift — and say whether recalibration or human takeover is the right response?
System / materials
Logged before/after segments from an approved plant or public sensor stream. Shift must be documented, not cherry-picked.
Expected failure modes
Calling noise “drift.” Retraining on the test period.
Done looks like
Drift chart, threshold for alert, and a written runbook line: recalibrate / degrade gracefully / human only.
Five C's
CT: drift vs. noise. CR: choosing a detection statistic. CO: peer verifies shift dates. CM: runbook one screen. CZ: who acts on a false calm.
Mentor role
Quality or reliability engineer reviews shift definition at Checkpoint 1 and runbook at Checkpoint 2. School-supervised.
Rubric calibration
R1: shift documented. R2: segments reproducible. R3: pre-shift baseline. R4: false alarm discussed. R5: runbook actionable. R6: refuses silent continue.
Two ways this goes wrong
(a) Retrain on everything — looks fixed, no detection story. (b) Drift claimed from three data points.
Credit lane fit
Lane A immediately. No verified credit claim.