Multimodal markers for technology-independent integration of augmented reality devices and surgical navigation systems
Mohamed Benmahdjoub, Wiro J. Niessen, Eppo B. Wolvius, Theo van Walsum
Abstract
Abstract Augmented reality (AR) permits the visualization of pre-operative data in the surgical field of view of the surgeon. This requires the alignment of the AR device’s coordinate system with the used navigation/tracking system. We propose a multimodal marker approach to align an AR device with a tracking system: in our implementation, an electromagnetic tracking system (EMTS). The solution makes use of a calibration method which determines the relationship between a 2D pattern detected by an RGB camera and an electromagnetic sensor of the EMTS. This allowed the projection of a 3D skull model on its physical counterpart. This projection was evaluated using a monocular camera and an optical see-through device (HoloLens 2) ( https://www.microsoft.com/en-us/hololens/ ) achieving an accuracy of less than 2.5 mm in the image plane of the HoloLens 2 (HL2). Additionally, 10 volunteers participated in a user study consisting of an alignment task of a pointer with 25 projections viewed through the HL2. The participants achieved a mean error of 2.7 1.3 mm and 2.9 2.9 $$^\circ $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mrow/> <mml:mo>∘</mml:mo> </mml:msup> </mml:math> in positional and orientation error. This study showcases the feasibility of the approach, provides an evaluation of the alignment, and finally, discusses its advantages and limitations.