Machine learning enhances assessment of proficiency in endovascular aortic repair simulations
Rebecca Andrea Conradsen Skov, Jonathan Lawaetz, Michael Strøm, Isabelle Van Herzeele, Lars Konge, Timothy Andrew Resch, Jonas Eiberg
Abstract
Endovascular skills can be learned through simulation-based education (SBE), and objective skills assessment is essential to ensure trainees' proficiency before proceeding to clinical procedures. Assessment requires trained instructors, which is a barrier to broader SBE implementation. This study aimed to evaluate if proficiency assessments in endovascular aortic repair (EVAR) can be automated using simulator-generated and biometric data combined with machine-learning. International EVAR novices followed a standardised SBE EVAR programme. Participants were assessed four times by experienced instructors using a validated assessment tool. Simulator performance metrics and biometric stress data were extracted from these cases to build prediction models based on contrasting groups' cut-off values and machine learning principles. During the program, 96 EVAR-cases were performed by 24 participants. Fifty (52%) cases were assessed as failed, and 46 (48%) as passed. The machine learning model outperformed the contrasting groups model with an accuracy of 80% compared to 65%. The area under the curve (AUC) of the ROC-curve for the machine learning model was 0.84 (95%CI: 0.75-0.94) compared to an AUC of 0.56 (95%CI: 0.48-0.64) in the contrasting groups model. The most predictive variables were total procedure time, activated clotting time, trainees' stress level, time to complete graft deployment, and heparin administration. Machine learning models based on simulator and biometric data can help determine proficiency in simulated EVAR procedures. This is a step toward minimizing the need for an instructor in basic EVAR SBE, but better simulator-generated metrics are needed to improve automated feedback.