Under the Cover Infant Pose Estimation Using Multimodal Data
Daniel G. Kyrollos, Anthony Fuller, Kim Greenwood, JoAnn Harrold, James R. Green
Abstract
Infant pose monitoring during sleep has multiple applications in both healthcare and home settings. In a healthcare setting, pose detection can be used for region-of-interest (ROI) detection and movement detection for noncontact-based monitoring systems. In a home setting, pose detection can be used to detect sleep positions, which has shown to have a strong influence on multiple health factors. However, pose monitoring during sleep is challenging due to heavy occlusions from blanket coverings and low lighting. To address this, we present a novel dataset, Simultaneously-collected multimodal Mannequin Lying pose (SMaL) dataset, for under-the-cover infant pose estimation. We collect depth and pressure imagery of an infant mannequin in different poses under various cover conditions. We successfully infer full body pose under the cover by training state-of-the-art pose estimation methods and leveraging existing multimodal adult pose datasets for transfer learning. We demonstrate a hierarchical pretraining strategy for transformer-based models to significantly improve performance on our dataset. Our best-performing model was able to detect joints under the cover within 25 mm 86% of the time with an overall mean error of 16.9 mm. Data, code, and models are publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/DanielKyr/SMaL</uri> .