Comparison of machine learning and human prediction to identify trauma patients in need of hemorrhage control resuscitation (ShockMatrix study): a prospective observational study
Tobias Gauss, Arthur James, Clélia Colas, Nathalie Delhaye, Mathilde Holleville, Benjamin Bijok, Marie Werner, Alain Meyer, Véronique Ramonda, Eric Cesareo, Hugues de Cherisey, Sofiane Medjkoune, Samia Salah, Jean‐Pierre Nadal, Jean-Denis Moyer, Antoine Vilotitch, Pierre Bouzat, Julie Josse, Julie Josse, Jeantrelle Caroline, Harrois Anatole, Raux Mathieu, Pasqueron Jean, Quesnel Christophe, Delhaye Nathalie, Godier Anne, Boutonnet Mathieu, Duranteau Olivier, Garrigue Delphine, Bourgeois Alexandre, Pottecher Julien, Tobias Gauss, Tobias Gauss, Etienne Montalescaut, Eric Meaudre, Jean-Luc Hanouz, Valentin Lefrancois, Gérard Audibert, Marc Leone, Emmanuelle Hammad, Gary Duclos, Vincent Legros, Thierry Floch, Pauline Perez, Anne-Claire Lukaszewicz, François-Xavier Jean, Véronique Ramonda, Thomas Geeraerts, Fanny Bounes, Jean Baptiste Bouillon, Benjamin Brieu, Sébastien Gettes, Nouchan Mellati, Jean-Stéphane David, Youri Yordanov, Leslie Dussau, Elisabeth Gaertner, Benjamin Popoff, Thomas Clavier, Perrine Lepêtre, Marion Scotto, Julie Rotival, Loan Malec, Claire Jaillette, Romain Mermillod Blondin, Etienne Escudier, Samuel Gay, Pierre Gosset, Clément Collard, Jean Pujo, Hatem Kallel, Alexis Fremery, Nicolas Higel, Mathieu Willig, Benjamin Cohen, Paer Selim Abback, Jerome Morel, Guillaume Bouhours
Abstract
Background: Machine learning could improve the timely identification of trauma patients in need of hemorrhage control resuscitation (HCR), but the real-life performance remains unknown. The ShockMatrix study aimed to compare the predictive performance of a machine learning algorithm with that of clinicians in identifying the need for HCR. Methods: Prospective, observational study in eight level-1 trauma centers. Upon receiving a prealert call, trauma clinicians in the resuscitation room entered nine predictor variables into a dedicated smartphone app and provided a subjective prediction of the need for HCR. These predictors matched those used in the machine learning model. The primary outcome, need for HCR, was defined as: transfusion in the resuscitation room, transfusion of more than four red blood cell units in 6 h of admission, any hemorrhage control procedure within 6 h, or death from hemorrhage within 24 h. The human and machine learning performances were assessed by sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and net clinical benefit. Human and machine learning agreement was assessed with Cohen's kappa coefficient. Findings: Between August 2022 and June 2024, out of 5550 potential eligible patients, 1292 were ultimately included in the analyses. The need for HCR occurred in 170/1292 patients (13%). The results showed a positive likelihood ratio of 3.74 (95% confidence interval [CI]: 3.20-4.36) and a negative likelihood ratio of 0.36 (95% CI: 0.29-0.46) for the human prediction and a positive likelihood ratio of 4.01 (95% CI: 3.43-4.70) and negative likelihood ratio of 0.35 (95% CI: 0.38-0.44) for the machine learning prediction. The combined use of human and machine learning prediction yielded a sensitivity of 83% (95% CI: 77-88%) and a specificity of 73% (95% CI: 70-75%). The Cohen's kappa coefficient showed an agreement of 0.51 (95% CI: 0.48-0.55). Interpretation: The prospective ShockMatrix temporal validation study suggests a comparable human and machine learning performance to predict the need for HCR using real-life and real-time information with a moderate level of agreement between the two. Machine learning enhanced decision awareness could potentially improve the detection of patients in need of HCR if used by clinicians. Funding: The study received no funding.