EEG2Text: Open Vocabulary EEG-to-Text Translation with Multi-View Transformer
Hanwen Liu, Daniel Hajialigol, Benny Antony, Aiguo Han, Xuan Wang
Abstract
Deciphering the intricacies of the human brain has captivated curiosity for centuries. Recent strides in Brain-Computer Interface (BCI) technology, particularly using motor imagery, have restored motor functions such as reaching, grasping, and walking in paralyzed individuals. However, unraveling natural language from brain signals remains a formidable challenge. Electroencephalography (EEG) is a non-invasive technique used to record electrical activity in the brain by placing electrodes on the scalp. Previous studies of EEG-to-text decoding have achieved high accuracy on small closed vocabularies, but still fall short of high accuracy when dealing with large open vocabularies. We propose a novel method, EEG2Text, to improve the accuracy of open vocabulary EEG-to-text decoding. Specifically, EEG2Text leverages EEG pre-training to enhance the learning of semantics from EEG signals and proposes a multi-view transformer to model the EEG signal processing by different t spatial regions of the brain. Experiments show that EEG2Text has superior performance, outperforming the state-of-the-art baseline methods by a large margin of up to 5% in absolute BLEU and ROUGE scores. EEG2Text shows great potential for a high-performance open-vocabulary brain-to-text system to facilitate communication. Our code is available at: https://github.com/ForeverNightmare/EEG2Text.