Autonomous System for EEG-Based Multiple Abnormal Mental States Classification Using Hybrid Deep Neural Networks Under Flight Environment
Dae-Hyeok Lee, Ji-Hoon Jeong, Baek-Woon Yu, Tae‐Eui Kam, Seong–Whan Lee
Abstract
Detection of the pilots’ mental states is particularly critical because their abnormal mental states (AbSs) could cause catastrophic accidents. In this study, we presented the feasibility of classifying the various specific AbSs (namely, low fatigue, high fatigue, low workload, high workload, low distraction, and high distraction) by applying the deep learning method. To the best of our knowledge, this study is the first attempt to classify multiple AbSs of pilots. We proposed the hybrid deep neural networks with five convolutional blocks and two long short-term memory layers for decoding multiple AbSs. We designed the model to extract the informative features from electroencephalography signals. A total of ten pilots conducted the experiment in a simulated flight environment. Compared with five conventional models, our proposed model achieved the highest grand-average accuracy of 68.04 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\pm$</tex-math> </inline-formula> 5.26)% which is at least 6.55% higher than other conventional models for classifying seven mental states across all subjects. Our proposed model could distinguish and classify low and high levels for each status category and give appropriate feedback to the subjects. In addition, we found nine indicators that showed the statistically significant differences between two mental states ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$<$</tex-math> </inline-formula> 0.05). Hence, we believe that it will contribute significantly to autonomous driving or autopilot advances based on artificial intelligence technology in the future.