New EEG Network Tracks Emotions Across Time and Space

Published in Brain Informatics, the study presents a spectral-spatial attention multi-temporal scale network that integrates spatial topology, spectral characteristics, and temporal dynamics to classify emotional states. The model achieved an average accuracy of 98.34% on the SEED dataset and 91.79% on the SEED-IV dataset, demonstrating competitive performance in discrete emotion classification tasks.

The development of SSA-MTSNet addresses specific limitations in existing EEG-based emotion recognition methods, which often struggle to capture the complex spatial-spectral dependencies and multi-temporal scale dynamics inherent in brain signals. Traditional approaches frequently treat EEG channels as independent time series, neglecting the intrinsic spatial dependencies among electrode sites. Furthermore, conventional convolutional neural networks often rely on two-dimensional convolutions that are inadequate for jointly modeling the spatio-temporal characteristics of EEG data. The proposed architecture aims to overcome these hurdles by preserving electrode topology while simultaneously enhancing signal frequencies and interactions among brain regions.

The core of the SSA-MTSNet architecture consists of three integrated components: a spectral-spatial attention module, a multi-temporal scale spatio-temporal convolution module, and a long short-term memory (LSTM) layer. The spectral-spatial attention module, inspired by the convolutional block attention module, operates in a serial fashion, first emphasizing informative frequency bands and then adjusting spatial attention weights across the electrode map. This mechanism allows the network to dynamically focus on the most discriminative spectral components and specific brain regions involved in emotional processing.

To capture temporal dynamics, the model employs a multi-temporal scale convolutional network with parallel branches that operate at different temporal resolutions. These branches extract local temporal dependencies at multiple scales, which are then aligned and fused through a progressive fusion process. Following this convolutional encoding, an LSTM layer models global temporal dependencies across the entire EEG sequence. This hierarchical temporal modeling strategy is central to the method, enabling the network to capture both short-term fluctuations and longer-range emotional dynamics within a unified framework.

The study evaluated the proposed model on three widely used public datasets: SEED, SEED-IV, and DEAP. The SEED dataset includes recordings from 15 participants who viewed movie clips categorized into positive, neutral, and negative emotions. The SEED-IV dataset also involves 15 subjects but categorizes emotional states into four classes: happy, sad, neutral, and fear. The DEAP dataset, used to assess generalization ability, consists of recordings from 32 participants who rated their emotional states in terms of valence and arousal after watching music video clips. EEG signals were preprocessed by extracting differential entropy features across five canonical frequency bands: delta, theta, alpha, beta, and gamma.

Experiments were conducted using tenfold cross-validation with strictly enforced subject-level data separation to prevent data leakage. The model, which contains 5.18 million trainable parameters, was trained using the PyTorch framework on an NVIDIA RTX 4060Ti GPU. The training process utilized the Adam optimizer with a learning rate of 1e-4 and a cross-entropy loss function. On the SEED dataset, the SSA-MTSNet achieved an average accuracy of 98.34% with a standard deviation of 1.67%. On the SEED-IV dataset, the model reached an average accuracy of 91.79% with a standard deviation of 1.58%. These results indicate that the model maintains high performance and stability across different experimental paradigms and emotion labeling strategies.

Ablation studies were performed to analyze the impact of different attention configurations and architectural components. Paired t-tests confirmed that the improvements of the spectral-to-spatial order over other configurations were statistically significant on the SEED dataset. The analysis also revealed that the inclusion of the fusion layer is critical for integrating multi-branch features, with its removal causing a substantial drop in accuracy, falling from 98.34% to 93.47% on the SEED dataset.

The generalization ability of the model was further evaluated on the DEAP dataset, which utilizes a different experimental paradigm based on music videos and continuous emotional dimensions. Under dataset-specific training settings, SSA-MTSNet achieved an accuracy of 95.13% for valence classification and 95.30% for arousal classification. These findings suggest that the proposed architecture can adapt to datasets with different electrode montages, data acquisition paradigms, and classification objectives, supporting its robustness for EEG-based emotion recognition.

Visualization of the attention weights provided neurophysiological insights into the model’s decision-making process. The attention weights were concentrated in the prefrontal regions, including electrodes AF3, FP1, FPZ, FP2, and AF4, and in the lateral temporal regions, such as F7, FT7, T7, and TP8. This spatial distribution aligns with established neurobiological knowledge, as the prefrontal cortex is involved in emotion evaluation and cognitive control, while temporal regions are associated with integrating emotional and perceptual information. The dynamic evolution of these weights over time indicates that the model effectively captures the temporally varying patterns of emotional states.

Compared with existing methods, SSA-MTSNet offers a favorable trade-off between accuracy, stability, and computational efficiency. While its accuracy is slightly lower than the top-ranked ACTNN model on some datasets, SSA-MTSNet demonstrates higher stability and the lowest inference latency among the compared methods, at approximately 0.40 ms per sample. This efficiency is significant for potential deployment in real-time affective computing applications. The study acknowledges limitations, noting that the evaluation was performed on public datasets under offline settings. Future work will focus on external validation, cross-dataset transfer evaluation, and lightweight optimization to enhance practical feasibility for resource-constrained devices.

The SEED dataset, used for primary evaluation, comprised recordings from 15 participants, including 7 males and 8 females, with a mean age of 23.27 years. Each subject completed three separate sessions, viewing 15 movie clips of approximately four minutes each, which were equally distributed across positive, neutral, and negative emotion categories. EEG signals were recorded using a 62-channel cap, down-sampled to 200 Hz, and filtered with a 0 to 75 Hz band-pass filter. The SEED-IV dataset also included 15 subjects, each participating in three independent sessions. In this dataset, each session comprised 24 movie clips of around two minutes each, categorized into four emotional states: happy, sad, neutral, and fear. EEG recordings for SEED-IV used the same 62-channel setup, down-sampled to 200 Hz, and filtered to a 1–75 Hz band-pass. The DEAP dataset, utilized to assess generalization, consisted of recordings from 32 participants, evenly split between males and females. Each subject watched 40 one-minute music video clips and rated their emotional states on a scale from 1 to 9 in terms of valence, arousal, liking, and dominance. EEG signals in DEAP were recorded using 32 channels according to the international 10/20 system, with original signals sampled at 512 Hz. Each trial in the DEAP dataset consisted of a 3-second pre-trial baseline and a 60-second stimulus segment.

Preprocessing of the EEG signals involved smoothing raw data to reduce noise artifacts. Continuous EEG data from the SEED and SEED-IV datasets were segmented using a sliding window of length t seconds, with overlapping regions introduced to enhance sample diversity and temporal continuity. An overlap factor determined the percentage of overlap between adjacent windows to maximize data utilization and prevent information loss, particularly when the total signal duration was not an integer multiple of the window length. Each segment of length t seconds was further divided into t non-overlapping 1-second slices. Differential entropy features were extracted from each slice across five canonical frequency bands: delta (1–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (30–50 Hz). Assuming the EEG signal follows a Gaussian distribution, differential entropy was calculated based on the mean and variance of the EEG segment. These features were then mapped into a two-dimensional matrix based on the relative spatial positions of the electrodes on the scalp to preserve spatial correlations. The differential entropy features of each frequency band were mapped to a 2D channel layout matrix, where each electrode’s corresponding position was filled with feature values and remaining positions were set to zero. The 2D matrices corresponding to different bands were stacked along a new dimension to form a 3D structure, which was then arranged in chronological order to form a 4D structure for model input.

The selection of EEG segment length significantly influenced emotion recognition performance. Experiments using a model pre-trained on 6-second segments revealed that shorter segments, such as 4 seconds, led to a substantial drop in average accuracy to 82.11% on SEED and 67.13% on SEED-IV. This degradation was attributed to insufficient spatio-temporal context within the shortened window, which hindered the model’s ability to capture the full temporal dynamics of emotional evolution. Extending the segment to 5 seconds improved accuracy to 95.88% on SEED and 90.22% on SEED-IV, suggesting that a modest increase in window length provided richer emotional cues. When the segment length was set to 6 seconds, the model achieved relatively stable performance, attaining 98.34% on SEED and 91.79% on SEED-IV. This length indicated an optimal balance between temporal resolution and contextual richness, supporting both sensitivity to short-term fluctuations and representation of long-term emotions. However, further increasing the segment length beyond 6 seconds resulted in declining performance on SEED, while SEED-IV showed modest gains. After exceeding 8 seconds, improvements became negligible, indicating that longer windows, while offering more contextual information, also introduced noise that could cause the model to rely excessively on global averages, thereby diminishing responsiveness to transient emotional variations.

The model’s computational efficiency was evaluated in comparison with existing methods. SSA-MTSNet contains 5.18 million trainable parameters and requires approximately 102.01 GFLOPs for a forward pass, occupying about 20.6 MB of parameter storage. While this parameter count is larger than most lightweight baselines, it is substantially smaller than heavy Transformer-based models such as Bi-ViTNet, which has approximately 170.00 million parameters. SSA-MTSNet attained the lowest inference latency among the compared methods, at approximately 0.40 ms per sample. This efficiency demonstrates that the increased parameter scale does not compromise deployment efficiency, offering a favorable trade-off among accuracy, stability, and inference speed. The training process was conducted for 60 epochs with a batch size of 64 and a dropout rate of 0.12 to reduce overfitting. The training time was approximately 30–40 seconds per epoch, while the average inference time was 0.42 ms per sample. These metrics highlight the model’s potential for practical application in affective computing systems where computational resources may be constrained.

Limitations of the study include the evaluation on public datasets under offline experimental settings. Although subject-level tenfold cross-validation was adopted to mitigate subject leakage, the evaluation was performed within the same datasets. The practical feasibility for resource-constrained devices, online systems, and broader affective computing applications remains to be further examined. Future work should focus on external validation, cross-dataset evaluation, lightweight optimization, and more rigorous interpretability analysis. The study suggests that the methodological feasibility observed in this study can be translated into stronger practical feasibility through these future efforts. The attention analysis provided interpretable insights, indicating that the model emphasizes neurophysiologically meaningful spatial patterns rather than arbitrary electrode combinations. Integrating multi-temporal scale convolution and LSTM enables the model to capture both short-term temporal variations and longer-range emotional dynamics, which is important because emotional processing evolves over time rather than occurring as an instantaneous neural event.

Subject of Research: Deep learning for EEG-based emotion recognition

Article Title: Emotion recognition using spectral-spatial attention multi-temporal scale network: EEG study

Article References: Tao, Z., Huang, G., Ma, L., Chen, Z., Liu, Y., & Hong, K. S. (2026). Emotion recognition using spectral-spatial attention multi-temporal scale network: EEG study. Brain Informatics, 13(1), Article 22. https://doi.org/10.1186/s40708-026-00309-x

Image Credits: AI Generated

DOI: 10.1186/s40708-026-00309-x

Keywords: EEG, emotion recognition, deep learning, neuroscience, affective computing, spectral-spatial attention, LSTM, SEED dataset, DEAP dataset, brain-computer interface, Emotion, recognition