Photoacoustics-Guided Real-Time Closed-Loop Control of Magnetic Microrobots Through Deep Learning
Richard Nauber, Johanna M. Hoppe, David Castellanos Robles, Mariana Medina‐Sánchez
Abstract
Medical microrobots promise to increase the efficacy and reduce the invasiveness of certain medical procedures in the future. Real-time tracking of the microrobot, actuation, and closed-loop control of its position under in vivo conditions is crucial to fulfill the task at hand. We present a system for closed-loop control of magnetic microrobots using dual-mode ultrasound and photoacoustic imaging. It employs GPU-accelerated beamforming and tracking to achieve real-time operation with a closed-loop cycle time of 100 ms. Artifacts from simultaneous imaging and magnetic actuation are suppressed through time-multiplexing. To address the challenge of detecting microrobots in low-contrast, strong-background images, we implemented real-time Deep Learning-based tracking. A custom dataset of various types of microrobots is curated from long-duration closed-loop control measurements and employed to fine-tune a pretrained detection model. We introduce a platform for real-time closed-loop control of microrobots and demonstrate its performance with a 300 μm spiral-shaped microrobot following a figure-of-8 shape under photoacoustic imaging guidance. The localization error is evaluated against an optical reference measurement. Our results show that photoacoustic-based tracking significantly outperforms ultrasound tracking, with the deep learning approach further reducing missed detections. This demonstrates the algorithm's ability to generalize to a previously unseen type of microrobot. We envision this platform to advance medical microrobotics research by providing real-time closed-loop control of untethered microrobots under deep tissue.