Deep Super-Resolution Network for rPPG Information Recovery and Noncontact Heart Rate Estimation
Zijie Yue, Shuai Ding, Shanlin Yang, Yang Hui, Zhili Li, Youtao Zhang, Yinghui Li
Abstract
Remote photoplethysmography (rPPG) enables non-contact heart rate (HR) estimation using facial videos and exhibits significant convenience over traditional contact-based HR measurement approaches. However, its effectiveness depends mainly on the number of pixels of the facial region; thus, the estimation accuracy is easily degraded due to missing rPPG information when the resolution of facial images is low. Given that in most real application scenarios, such as group-oriented and long-distance vital signs measurement, only low-resolution facial videos are available, finding a solution to recover rPPG information is a key to improve the performance of non-contact HR measurement. In this paper, we propose a two-stage deep learning scheme to achieve accurate HR measurement from low-resolution facial videos. In the first stage, an rPPG information recovery network is proposed to recover the rPPG information during video super-resolution (SR) processing. The up-scaled images are then fed into a temporally-aware HR measurement network in the second stage. An attention mechanism that reassigns the weights of the temporal information in each feature channel is designed to improve the measurement accuracy. The mean absolute percentage error achieves 3.320 and 4.231, while the Pearson’s correlation coefficient reaches 0.930 and 0.892 on two datasets respectively. Experimental results show that rPPG information could be effectively recovered through SR processing. Compared with the state-of-the-art approaches that treat high-resolution images as the input, the proposed method achieved closely comparable measurement performance using merely low-resolution images.