Temporal validation of machine learning models for pre-eclampsia prediction using routinely collected maternal characteristics: A validation study

Abstract

BACKGROUND: Pre-eclampsia (PE) contributes to more than one-fourth of all maternal deaths and half a million newborn deaths worldwide every year. Early screening and interventions can reduce PE incidence and related complications. We aim to 1) temporally validate three existing models (two machine learning (ML) and one logistic regression) developed in the same region and 2) compare the performances of the validated ML models with the logistic regression model in PE prediction. This work addresses a gap in the literature by undertaking a comprehensive evaluation of existing risk prediction models, which is an important step to advancing this field. METHODS: We obtained a dataset of routinely collected antenatal data from three maternity hospitals in South-East Melbourne, Australia, extracted between July 2021 and December 2022. We temporally validated three existing models: extreme gradient boosting (XGBoost, 'model 1'), random forest ('model 2') ML models, and a logistic regression model ('model 3'). Area under the receiver-operating characteristic (ROC) curve (AUC) was evaluated discrimination performance, and calibration was assessed. The AUCs were compared using the 'bootstrapping' test. RESULTS: The temporal evaluation dataset consisted of 12,549 singleton pregnancies, of which 431 (3.43 %, 95 % confidence interval (CI) 3.13-3.77) developed PE. The characteristics of the temporal evaluation dataset were similar to the original development dataset. The XGBoost 'model 1' and the logistic regression 'model 3' exhibited similar discrimination performance with an AUC of 0.75 (95 % CI 0.73-0.78) and 0.76 (95 % CI 0.74-0.78), respectively. The random forest 'model 2' showed a discrimination performance of AUC 0.71 (95 % CI 0.69-0.74). Model 3 showed perfect calibration performance with a slope of 1.02 (95 % CI 0.92-1.12). Models 1 and 2 showed a calibration slope of 1.15 (95 % CI 1.03-1.28) and 0.62 (95 % CI 0.54-0.70), respectively. Compared to the original development models, the temporally validated models 1 and 3 showed stable discrimination performance, whereas model 2 showed significantly lower discrimination performance. Models 1 and 3 showed better clinical net benefits between 3 % and 22 % threshold probabilities than default strategies. CONCLUSIONS: During temporal validation of PE prediction models, logistic regression and XGBoost models exhibited stable prediction performance; however, both ML models did not outperform the logistic regression model. To facilitate insights into interpretability and deployment, the logistic regression model could be integrated into routine practice as a first-step in a two-stage screening approach to identify a higher-risk woman for further second stage screening with a more accurate test.

Topics & Concepts