Spiking Neural Networks Enhance Multimodal EEG and fMRI Brain State Decoding

Biologically inspired spiking neural network frameworks integrating simultaneous electroencephalography and functional magnetic resonance imaging data have demonstrated superior performance over unimodal approaches in decoding resting-state brain states. In brain informatics, multimodal data integration captures complementary spatial and temporal characteristics of neural activity to generate more accurate and biologically meaningful representations of brain function. However, combining established measurement modalities such as EEG and fMRI presents significant challenges due to fundamental heterogeneity in their temporal and spatial feature representations. To address these challenges, researchers developed spiking neural network frameworks employing early, intermediate, and late fusion strategies, featuring a novel ensemble-based explainable late-fusion classifier designated as X-Meta.

Methodologically, the design extends spiking neural network reservoir architectures to support multimodal learning across separate reservoirs for EEG and fMRI, enabling the direct modelling of cross-modal interactions. The fusion frameworks were evaluated on a resting-state simultaneous EEG-fMRI dataset to decode Eyes-Closed and Eyes-Open-Eyes-Closed brain states, benchmarking results against deep learning models and conventional machine learning utilizing alpha alone and multiple frequency bands. The study also examined two distinct EEG feature representations—raw amplitude and Power Spectral Density—integrated with fMRI to assess how input representations influence multimodal fusion efficacy.

The experimental results demonstrated that all evaluated fusion strategies significantly outperformed unimodal fMRI, achieving accuracy improvements exceeding 6.6% to 18%. Performance gains relative to the EEG modality depended on the feature representation: for Power Spectral Density representations, post-reservoir fusion strategies enhanced accuracy by 2.1% to 4.21%, whereas raw amplitude representations showed smaller gains of approximately 1.3% with intermediate fusion and X-Meta, reflecting differences in the relative discriminative strength of each format. Additionally, X-Meta delivered model-level feature attribution, identifying EEG and fMRI features associated with previously documented functional brain regions while highlighting exploratory candidate markers to provide interpretable insights into the network’s classification decisions.

Subject of Research: Multimodal EEG and fMRI data integration using spiking neural networks

Article Title: Brain-inspired spiking neural network frameworks for multimodal spatiotemporal brain data integration: a case study on EEG-fMRI data

Article References: Shahab, N., Doborjeh, M., Kasabov, N., Shahab, I., & Doborjeh, Z. (2026). Brain-inspired spiking neural network frameworks for multimodal spatiotemporal brain data integration: a case study on EEG-fMRI data. Brain Informatics. https://doi.org/10.1186/s40708-026-00340-y

Image Credits: AI Generated

DOI: 10.1186/s40708-026-00340-y

Keywords: spiking neural networks, EEG-fMRI integration, multimodal neuroimaging, brain informatics, brain state decoding, machine learning, feature attribution, Brain-inspired, spiking, neural, network, frameworks