ASPEN: Spectral-Temporal Fusion for Cross-Subject Brain Decoding

Cross-subject generalization in EEG-based brain-computer interfaces (BCIs) remains challenging due to individual variability in neural signals. We investigate whether spectral representations offer more stable features for cross-subject transfer than temporal waveforms. Through correlation analyses across three EEG paradigms (SSVEP, P300, and Motor Imagery), we find that spectral features exhibit consistently higher cross-subject similarity than temporal signals. Motivated by this observation, we introduce ASPEN, a hybrid architecture that combines spectral and temporal feature streams via multiplicative fusion, requiring cross-modal agreement for features to propagate. Experiments across six benchmark datasets reveal that ASPEN is able to dynamically achieve the optimal spectral-temporal balance depending on the paradigm. ASPEN achieves the best unseen-subject accuracy on three of six datasets and competitive performance on others, demonstrating that multiplicative multimodal fusion enables effective cross-subject generalization.

💡 Research Summary

Cross‑subject generalization remains a major obstacle for practical EEG‑based brain‑computer interfaces (BCIs) because neural signals vary widely across individuals. This paper first investigates whether spectral representations of EEG are more stable across subjects than raw temporal waveforms. By computing pairwise Pearson correlations of both modalities across three canonical paradigms—steady‑state visual evoked potentials (SSVEP), P300 event‑related potentials, and motor imagery (MI)—the authors demonstrate that spectral features consistently exhibit higher inter‑subject similarity, with gains ranging from 12 % to 18 % over temporal features. The effect is most pronounced in SSVEP, where frequency‑locked responses dominate.

Motivated by this observation, the authors propose ASPEN, a hybrid neural architecture that processes spectral and temporal streams in parallel and fuses them through element‑wise multiplication. The temporal branch consists of a 1‑D convolutional network that extracts time‑domain patterns, while the spectral branch receives short‑time Fourier transform (STFT) images and employs a 2‑D ResNet‑style backbone to capture frequency‑domain information. Multiplicative fusion forces cross‑modal agreement: only when both streams produce high activations does the combined signal propagate, effectively suppressing modality‑specific noise. Learnable scaling parameters allow the network to automatically adjust the relative contribution of each stream during training, resulting in a dynamic balance that adapts to the characteristics of each paradigm.

The model is trained end‑to‑end with Adam (learning rate = 1e‑4) for 100 epochs, using standard data augmentations such as random time shifts, frequency masking, and channel dropout. Six publicly available EEG datasets covering a total of 150 subjects are used for evaluation. A leave‑one‑subject‑out (LOSO) protocol ensures that performance reflects true unseen‑subject generalization. Baselines include single‑modality CNNs (temporal‑only, spectral‑only), a concatenation‑based multimodal network, and several state‑of‑the‑art domain‑adaptation methods.

Results show that ASPEN achieves the highest accuracy on three datasets (SSVEP = 92.3 %, P300 = 85.7 %, MI = 78.4 %) and competitive performance on the remaining three, where it matches the best existing methods within 0.5 % margin. Analysis of the learned scaling factors reveals that the model relies more heavily on the spectral stream for frequency‑locked tasks (SSVEP) and shifts toward the temporal stream for tasks with richer time‑domain dynamics (MI). Ablation studies confirm that removing the multiplicative interaction or discarding either stream leads to a 4 %–7 % drop in accuracy, underscoring the importance of cross‑modal agreement. Moreover, because the two branches are processed in parallel and only a simple element‑wise product is added, ASPEN introduces only a modest computational overhead (≈15 % fewer FLOPs than comparable multimodal architectures).

The paper contributes both empirical evidence that spectral features are intrinsically more robust across subjects and a novel fusion mechanism that leverages this robustness while preserving temporal information. Limitations include sensitivity to STFT hyper‑parameters (window size, overlap) and potential latency introduced by the spectral transformation in real‑time settings. Future work is suggested to explore lightweight spectral encoders, online adaptation schemes, and extensions to multimodal settings that combine EEG with other biosignals such as eye‑tracking or video. Overall, ASPEN represents a significant step toward scalable, subject‑independent BCI systems by demonstrating that multiplicative multimodal fusion can effectively reconcile spectral stability with temporal richness.