Clinical prediction models are often validated with aggregate metrics like AUROC, but a model that looks stable can still shift its behaviour at the bedside. A new arXiv preprint argues that in ICU sepsis prediction, the choice of missing-data imputation can change which patients receive an alert even when the AUROC stays roughly the same. The authors describe this as an under-measured reliability risk in clinical prediction.
Rather than relying on reconstruction error or broad model performance, the paper proposes evaluating imputation strategies through alert-list concordance — whether the same patients would be flagged under different imputation approaches. This moves the focus from statistical fit to operational consistency, which matters more for clinicians responding to alerts.
Since the work is a preprint and not yet peer-reviewed, it should be read as a methodological proposal rather than a settled benchmark. Still, it highlights a practical gap in how imputation quality is currently assessed in clinical machine learning.