Markerless Motion Tracking With Noisy Video and IMU Data
Soyong Shin, LI Zhi-xiong, Eni Halilaj
Abstract
Objective:Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home.Methods:We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities.Results:On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean$\pm$std root-mean-squared errors by 7.6$\pm$5.4$^{\circ }$and 5.9$\pm$3.3$^{\circ }$, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0$\pm$5.3$^{\circ }$for sensor-misplacement errors of up to 30.0$\pm$13.7$^{\circ }$and by up to 7.4$\pm$5.5$^{\circ }$for joint-center-estimation errors of up to 101.1$\pm$11.2 mm, across joints.Conclusion:These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.