Machine learning for developing a prediction model of hospital admission of emergency department patients: Hype or hope?
Anne de Hond, Wouter Raven, Laurens Schinkelshoek, Menno I. Gaakeer, Ewoud ter Avest, Özcan Sir, Heleen Lameijer, Roger A.P.A. Hessels, Resi Reijnen, Evert de Jonge, Ewout W. Steyerberg, Christian H. Nickel, Bas de Groot
Abstract
OBJECTIVE: Early identification of emergency department (ED) patients who need hospitalization is essential for quality of care and patient safety. We aimed to compare machine learning (ML) models predicting the hospitalization of ED patients and conventional regression techniques at three points in time after ED registration. METHODS: We analyzed consecutive ED patients of three hospitals using the Netherlands Emergency Department Evaluation Database (NEED). We developed prediction models for hospitalization using an increasing number of data available at triage, ∼30 min (including vital signs) and ∼2 h (including laboratory tests) after ED registration, using ML (random forest, gradient boosted decision trees, deep neural networks) and multivariable logistic regression analysis (including spline transformations for continuous predictors). Demographics, urgency, presenting complaints, disease severity and proxies for comorbidity, and complexity were used as covariates. We compared the performance using the area under the ROC curve in independent validation sets from each hospital. RESULTS: We included 172,104 ED patients of whom 66,782 (39 %) were hospitalized. The AUC of the multivariable logistic regression model was 0.82 (0.78-0.86) at triage, 0.84 (0.81-0.86) at ∼30 min and 0.83 (0.75-0.92) after ∼2 h. The best performing ML model over time was the gradient boosted decision trees model with an AUC of 0.84 (0.77-0.88) at triage, 0.86 (0.82-0.89) at ∼30 min and 0.86 (0.74-0.93) after ∼2 h. CONCLUSIONS: Our study showed that machine learning models had an excellent but similar predictive performance as the logistic regression model for predicting hospital admission. In comparison to the 30-min model, the 2-h model did not show a performance improvement. After further validation, these prediction models could support management decisions by real-time feedback to medical personal.