Midas++: Generating Training Data of mmWave Radars From Videos for Privacy-Preserving Human Sensing With Mobility
Kaikai Deng, Dong Zhao, Zihan Zhang, Shuyue Wang, Wenxin Zheng, Huadóng Ma
Abstract
Millimeter wave radar is gaining traction recently for enabling privacy-preserving human sensing. However, the lack of large-scale, dynamic radar datasets impedes progress in developing robust and generalized deep learning models for mobile sensing applications. To address this problem, we resort to designing a software pipeline that leverages wealthy dynamic videos to generate synthetic radar data, but it faces two key challenges including i) incorrect camera and human positions leading to erroneous superposition of signal intensity and ii) the signal reflection of the background and humans in mobile scenes. To this end, we design <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Midas++</monospace> to utilize rich videos to generate realistic radar data via two components: (i) a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">human mesh fitting and calibration</i> component calculates the camera ego-motion parameters to calibrate the extracted human positions; (ii) a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">reflection and noise signal estimation</i> component combines several key modules, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">depth prediction</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">reflection model</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">spatiotemporal noise estimation</i> , to output coarse radar data, followed by a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U-Net</i> model to generate realistic radar data. We implement and evaluate <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Midas++</monospace> with video data from public data sources and real-world radar data, demonstrating that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Midas++</monospace> outperforms other state-of-the-art approaches for both activity recognition and object detection tasks.