External validation, impact assessment and clinical utilization of clinical prediction models: a prospective cohort study
Banafsheh Arshi, Laura Cowley, Eline Rijnhart, Kelly Reeve, Luc Smits, Laure Wynants
Abstract
OBJECTIVES: We aimed to assess paths taken by clinical prediction models (CPMs) after development by quantifying external validation, impact assessment, and utilization in clinical practice. STUDY DESIGN AND SETTING: We followed a random sample of 109 regression-based CPM development articles published between 1995 and 2020 by performing a forward citation search. We estimated 5- and 10-year probabilities of validation and impact assessment after development of CPMs using Kaplan-Meier analysis. In addition, we conducted a survey among the authors of the development articles to determine whether the CPMs had been used in clinical settings. RESULTS: Eighteen (17%) CPM development articles reported a CPM that was externally validated after development. Five- and 10-year probabilities of validation were 0.13 (0.06-0.19) and 0.16 (0.08-0.23), respectively. Only 1 article had a CPM with impact assessment during follow-up (10-year probability: 0.01 [0-0.04]). Among the 34 (31%) articles with a survey response, 17 (50%) had CPMs that had been used in clinical practice, in a median of five sites (interquartile range: 1-347). Of these models, only 4 (24%) were externally validated, and none had undergone impact assessment. CONCLUSION: Despite evidence of utilization in clinical settings, few models are externally validated after development, and published impact assessment is scarce. To prevent compromising patient safety, it is crucial to intensify efforts to promote external validation and impact assessment of prediction models.