Deep Learning-Based Multimodal Abnormal Gait Classification Using a 3D Skeleton and Plantar Foot Pressure

Abstract

Classification of pathological gaits has an important role in finding a weakened body part or disease and supporting a doctor’s decision. Many machine learning-based approaches have been proposed that automatically classify abnormal gait patterns using various sensors, such as inertial sensors, depth cameras and foot pressure plates. In this paper, we present a deep learning-based abnormal gait classification method employing both a 3D skeleton (obtained with a depth camera) and plantar foot pressure. We collected skeleton and foot pressure data simultaneously for 1 normal and 5 pathological gaits (antalgic, lurching, steppage, stiff-legged, and Trendelenburg gaits) and classified the gaits by using single-modal models fed either skeleton or pressure data and a multimodal model fed both data types together. In the proposed method, we fed the sequential skeleton and average foot pressure data into recurrent neural network (RNN)-based encoding layers and convolutional neural network (CNN)-based encoding layers, respectively. Finally, the output features were concatenated and fed to the classification layers. The pressure-based and skeleton-based single-modal models achieved classification accuracies of 68.82% and 93.40%, respectively. The proposed multimodal hybrid model using skeleton and foot pressure data together showed improved performance, with an accuracy of 95.66%. We fine-tuned the hybrid model by applying a 3-step training methodology and ultimately increased the accuracy to 97.60%. This study indicates that the integrated features of the skeleton and foot pressure data represent both the spatiotemporal motion information and weight distribution, so data fusion can generate a positive effect in pathological gait classification.

Topics & Concepts