V-DixMatch: A Semi-Supervised Learning Method for Human Action Recognition in Night Video Sensing

Abstract

Human Action Recognition (HAR) in night video sensing has become a crucial task for a wide range of applications (e.g., night surveillance and self-driving at night). Recently, Fully Supervised Learning (FSL) methods by training with large-scale labeled data achieve substantial performance in HAR. However, since the data captured by red–green–blue (RGB) sensor-based cameras in the night (low-light) scenes suffer from poor illumination, it leads to difficult annotations and limits the applications of night HAR by the FSL method. A potential solution for this issue is to transfer knowledge learned from normal-light videos to low-light ones using Semi-Supervised Learning (SSL) methods, such as Unsupervised Domain Adaptation (UDA) and self-training. Although, these SSL methods have shown promising results in datasets with the same or similar data distributions, they struggle with the dataset with large domain discrepancies, e.g., normal-light and low-light conditions. To address this issue, a new SSL method called V-DixMatch, which includes the pixel-level adaptation, the feature-level adaptation, and the Cross-Domain Video-based (CDV) self-training, is proposed in this work. Specifically, the pixel-level adaptation reduces the domain discrepancy in low-level features by pixel-level processing and the feature-level adaptation approximates domain discrepancy in high-level features through adversarial learning. Then, CDV self-training further provides a robust self-training strategy by designing a video-based Blended Frames Augmentation (BFA), which is a data augmentation method tailored for video-based data, and a pseudo-label collection, which improves the quality of pseudo-labels in self-training. Extensive experimental results demonstrate that V-DixMatch achieves state-of-the-art performance in the SSL low-light HAR, and is even comparable to some FSL methods.

Topics & Concepts