Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)
Abhijit Mishra, Shreya Shukla, José Sulla-Torres, Jacek Gwizdka, Shounak Roychowdhury
Abstract
Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI.This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal.The approach involves three stages:(1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference.Experiments on a public EEG dataset collected for six subjects with image stimuli and text captions demonstrate the efficacy of multimodal LLMs (LLAMA-V3, MISTRAL-V0.3,QWEN2.5),validated using traditional language generation evaluation metrics, as well as fluency and adequacy measures.This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing.