Multimodal fNIRS–EEG sensor fusion: Review of data-driven methods and perspective for naturalistic brain imaging
Tomás Codina, Benjamin Blankertz, Alexander von Lühmann
Abstract
Functional near-infrared spectroscopy (fNIRS), high-density diffuse optical tomography (HD-DOT), and electroencephalography (EEG) are established, cost-effective, and non-invasive neuroimaging techniques, whose integration represents a promising direction for brain activity decoding with high spatiotemporal resolution in naturalistic scenarios. However, robust machine-learning methods for combining these signals remain challenging. In this review, we focus on multimodal fusion methods, emphasizing data-driven unsupervised symmetric techniques, and study their performance on our own HD-fNIRS-EEG data with synthetic ground truth. To this end, we performed a systematic method-oriented survey on fNIRS/DOT-EEG fusion, categorizing works based on fusion strategies, and identifying common artifact removal techniques and integrated auxiliary signals. Our review indicates that while many studies incorporate robust artifact handling for EEG, confounder correction in fNIRS remains limited to filtering or motion removal. Moreover, short-separation measurements and other auxiliary signals for fNIRS remain underutilized. Fusion methods predominantly rely on data concatenation, model-based, or decision-level strategies, while source-decomposition techniques are underrepresented, despite their potential for revealing more complex latent neurovascular coupling processes. To address the scarcity of multimodal public datasets, we generated a realistic synthetic HD-fNIRS-EEG dataset that simulates a finger tapping motor task, with concurrent suppression of EEG alpha-band power and an increase in hemoglobin in fNIRS from a shared neuronal source. We illustrate a proof-of-concept comparison of some source-decomposition methods on this dataset and provide the full implementations and an example Jupyter notebook to reproduce and extend these results.