IAR-Net: A Human–Object Context Guided Action Recognition Network for Industrial Environment Monitoring
Naval Kishore Mehta, Shyam Sunder Prasad, Sumeet Saurav, R Saini, Sanjay Singh
Abstract
Industry 5.0 and increased industrial automation have driven the demand for systems recognizing human activities in industrial environments. Vision-based systems for human activity recognition at industrial sites may be helpful in ergonomic studies. Besides, these systems may help identify possible deviations in assembly line standard operating procedures (SOPs) and facilitate early rejection of items. The primary challenge these systems face is the limited research dedicated to accurately comprehending and interpreting human actions and intentions within the intricate and dynamic contexts of industrial settings. While human action recognition (HAR) has seen significant exploration in machine learning, its application within industrial settings remains relatively unexplored. Besides, the current research lacks a realistic open-source dataset accurately representing human actions in industrial settings. To this end, this article first introduces the Lathe-operator activity monitoring in industrial surroundings (LAMIS) database, a specialized database that covers 17 categories of industrial-like actions. Second, the article presents the design and implementation of a novel deep learning architecture called industrial action recognition network (IAR-Net) for industrial activity recognition. IAR-Net uses RGB spatiotemporal cues to capture human context and granularities. We trained the proposed IAR-Net using a novel adaptive frame sampling approach that adaptively selects keyframes from video clips, reducing the overall computational cost. The IAR-Net model achieves a baseline recognition accuracy of 85.39% on the in-house LAMIS database. In addition, the model delivered a state-of-the-art accuracy of 95.23% on the related benchmark HRI30 dataset. The results demonstrate the efficacy of the proposed IAR-Net model and its adaptivity in several human activity recognition tasks in industrial settings.