ASAP: A Semi-Autonomous Precise System for Telesurgery During Communication Delays
Glebys Gonzalez, Mythra V. Balakuntala, Mridul Agarwal, Thomas Low, Bruce Knoth, Andrew W. Kirkpatrick, Jessica McKee, Gregory D. Hager, Vaneet Aggarwal, Yexiang Xue, Richard M. Voyles, Juan Wachs
Abstract
In remote, rural, and disadvantaged areas, telesurgery can be severely hindered by limitations of communication infrastructure. In conventional telesurgery, delays as small as 300ms can produce fatal surgical errors. To mitigate the effect of communication delays during telesurgery, we introduce a semi-autonomous system that decouples the user interaction from the robot execution. This system uses a physics-based simulator where a surgeon can demonstrate individual surgical subtasks, with immediate graphical feedback. Each subtask is performed asynchronously, unaffected by communication latency, jitter, and packet loss. A surgical step recognition module extracts the intended actions from the observed surgeon-simulation interaction. The remote robot can perform each one of these actions autonomously. The action recognition system leveraged a transfer learning approach that minimized the data needed during training, and most of the learning is obtained from simulated data. We tested this system in two tasks: fluid-submerged peg transfer (resembling bleeding events) and surgical debridement. The system showed robustness to delays of up to 5 seconds, maintaining a performance rate of 87% for peg transfer and 88% for debridement. Also, the framework reduced the completion time under delays by 45% and 11% during peg transfer and debridement, respectively.