Automated recognition of construction worker activities using multimodal decision-level fusion

Abstract

This paper proposes an automated approach for construction worker activity recognition by integrating video and acceleration data, employing a decision-level fusion method that combines classification results from each data modality using the Dempster-Shafer Theory (DS). To address uneven sensor reliability, the Category-wise Weighted Dempster-Shafer (CWDS) approach is further proposed, estimating category-wise weights during training and embedding them into the fusion process. An experimental study with ten participants performing eight construction activities showed that models trained using DS and CWDS outperformed single-modal approaches, achieving accuracies of 91.8% and 95.6%, about 7% and 10% higher than those of vision-based and acceleration-based models, respectively. Category-wise improvements were also observed, indicating that the proposed multimodal fusion approaches result in a more robust and balanced model. These results highlight the effectiveness of integrating vision and accelerometer data through decision-level fusion to reduce uncertainty in multimodal data and leverage the strengths of single sensor-based approaches. • Decision-level fusion improves performance in construction activity recognition. • Dempster-Shafer method fuses vision and acceleration data at the decision level. • Category-specific weights balance model reliability from different sensor sources. • Fusion method leverages category-specific strengths of each sensor sources.

Topics & Concepts