External Validation of a Commercial Acute Kidney Injury Predictive Model
Sayon Dutta, Dustin McEvoy, Lisette Dunham, Ronelle Stevens, David Rubins, Gearoid M. McMahon, Lipika Samal
Abstract
BackgroundHospital-acquired acute kidney injury (HA-AKI), a common complication in hospitalized patients that increases morbidity and mortality, is challenging to predict given its multifactorial etiology. This study evaluated the performance of a commercial machine learning model developed by Epic Systems Corporation to predict the risk of developing HA-AKI in adult emergency department and hospitalized patients at a large health care system. MethodsThe Epic Risk of HA-AKI predictive model is a gradient-boosted forest ensemble that evaluates demographic characteristics, comorbidities, medication administration, and other clinical variables. The prospectively implemented model generated predictions hourly. Encounter-level performance and prediction-level model performance were evaluated by using the area under the receiver operating curve (AUROC) and the area under the precision recall curve (AUPRC) metrics. Net benefit was evaluated by using decision curve analysis. Test characteristics and lead time warning were also evaluated. The study included patients with at least two serum creatinine measurements and no history of stage 4 or 5 chronic kidney disease or end-stage renal disease between August 2022 and January 2023. ResultsDuring a 5-month period, 39,891 encounters were evaluated. The incidence of the primary outcome — development of Kidney Disease: Improving Global Outcomes stage 1 HA-AKI during the encounter — was 24.5%. The encounter-level AUROC was 0.77 (95% confidence interval [CI], 0.76 to 0.78), and the AUPRC was 0.49 (95% CI, 0.48 to 0.50). With a prediction horizon of 48 hours, the AUROC was 0.76 (95% CI, 0.76 to 0.76), and the AUPRC was 0.19 (95% CI, 0.19 to 0.19). At a score threshold of 50, the positive predictive value was 88%, sensitivity was 50%, and median lead-time warning was 21.6 hours before stage 1 HA-AKI occurred. ConclusionsThe Epic Risk of HA-AKI predictive model performed moderately well. Additional study is required to determine its clinical impact.