Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language
Chen Feng, L. Cao, Di Wu, En Zhang, Ting Wang, Xiaowei Jiang, Jinbo Chen, Hui Wu, Siyu Lin, Qiming Hou, Junming Zhu, Jie Yang, Mohamad Sawan, Yue Zhang
Abstract
Recent advances in brain-computer interfaces (BCIs) have demonstrated the potential to decode language from brain activity into sound or text, which has predominantly focused on alphabetic languages, such as English. However, logosyllabic languages, such as Mandarin Chinese, present marked challenges for establishing decoders that cover all characters, due to its unique syllable structures, extended character sets (e.g., over 50,000 characters for Mandarin Chinese), and complex mappings between characters and syllables, thus hindering practical applications. Here, we leverage the acoustic features of Mandarin Chinese syllables, constructing prediction models for syllable components (initials, tones, and finals), and decode speech-related stereoelectroencephalography (sEEG) signals into coherent Chinese sentences. The results demonstrate a high sentence-level offline decoding performance with a median character accuracy of 71.00% over the full spectrum of characters in the best participant. We also verified that incorporating acoustic-related features into the design of prediction models substantially enhances the accuracy of initials, tones, and finals. Moreover, our findings revealed that effective speech decoding also involves subcortical structures like the thalamus in addition to traditional language-related brain regions. Overall, we established a brain-to-sentence decoder for logosyllabic languages over full character set with a large intracranial electroencephalography dataset.