Integrating Language-Image Prior into EEG Decoding for Cross-Task Zero-Calibration RSVP-BCI

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interface (BCI) is an effective technology used for information detection by detecting Event-Related Potentials (ERPs). The current RSVP decoding methods can perform well in decoding EEG signals within a single RSVP task, but their decoding performance significantly decreases when directly applied to different RSVP tasks without calibration data from the new tasks. This limits the rapid and efficient deployment of RSVP-BCI systems for detecting different categories of targets in various scenarios. To overcome this limitation, this study aims to enhance the cross-task zero-calibration RSVP decoding performance. First, we design three distinct RSVP tasks for target image retrieval and build an open-source dataset containing EEG signals and corresponding stimulus images. Then we propose an EEG with Language-Image Prior fusion Transformer (ELIPformer) for cross-task zero-calibration RSVP decoding. Specifically, we propose a prompt encoder based on the language-image pre-trained model to extract language-image features from task-specific prompts and stimulus images as prior knowledge for enhancing EEG decoding. A cross bidirectional attention mechanism is also adopted to facilitate the effective feature fusion and alignment between the EEG and language-image features. Extensive experiments demonstrate that the proposed model achieves superior performance in cross-task zero-calibration RSVP decoding, which promotes the RSVP-BCI system from research to practical application.

💡 Research Summary

**
Rapid Serial Visual Presentation (RSVP)–based brain‑computer interfaces (BCIs) detect target images by identifying the P300 event‑related potential (ERP) that appears when a rare target stimulus is presented among a rapid stream of images. While many decoding methods achieve high accuracy when trained and tested on the same RSVP task (subject‑dependent decoding) or on the same task with data from many subjects (zero‑calibration decoding), their performance drops dramatically when the model trained on one task is applied to a different task without any calibration data. This “cross‑task zero‑calibration” problem limits the practical deployment of RSVP‑BCI systems for diverse real‑world scenarios where the target category and visual context change.

To address this gap, the authors first constructed an open‑source dataset named “NeuBCI Target Retrieval RSVP‑EEG Dataset”. The dataset contains EEG recordings synchronized with the exact stimulus images for three distinct target‑retrieval tasks: identifying planes in satellite images, cars in drone footage, and people in street scenes. A total of 71 participants were recruited (20 for the plane task, 20 for the car task, and 31 for the people task), with no overlap among tasks. All subjects had normal vision, no neurological disorders, and gave informed consent. The dataset, together with the stimulus image sequences, is publicly available (DOI:10.57760/sciencedb.14812), providing a rare resource that couples EEG signals with their corresponding visual inputs across multiple tasks.

The core contribution is the EEG with Language‑Image Prior fusion Transformer (ELIPformer), a novel architecture that fuses EEG data with multimodal prior knowledge extracted from a large‑scale language‑image pre‑training model (CLIP). ELIPformer consists of four main components:

1. EEG Encoder – a Transformer‑based module that tokenizes the multichannel EEG time series and captures global temporal‑spatial patterns. Unlike conventional CNNs, the self‑attention mechanism enables the model to learn long‑range dependencies across electrodes and time points.

2. Prompt Encoder – built on CLIP, this module receives two inputs: (a) a task‑specific textual prompt (e.g., “plane”, “car”, “people”) and (b) the visual stimulus image presented at each time step. CLIP jointly encodes the text and image into a shared embedding space, producing a language‑image feature vector that reflects both semantic meaning and visual appearance.

3. Cross Bidirectional Attention (Bi‑Attention) Module – a symmetric attention block that lets EEG tokens attend to language‑image tokens and vice‑versa. This mechanism aligns the two modalities, allowing the EEG representation to be enriched with semantic cues (e.g., the concept of “plane”) while the visual embedding is informed by the neural response patterns.

4. Metric‑Learning Classification Head – the fused representation is fed into a metric‑learning loss (e.g., contrastive or triplet loss) that explicitly pushes target‑related embeddings closer together and separates them from non‑target embeddings, improving discriminability for P300 detection.

The authors conducted extensive experiments under two regimes:

• In‑Task Zero‑Calibration – training and testing on the same task but on unseen subjects. ELIPformer outperformed state‑of‑the‑art zero‑calibration models such as TFF‑Former and EEGConformer by 3–5 percentage points in accuracy, confirming that the multimodal prior does not hurt within‑task performance.

• Cross‑Task Zero‑Calibration – training on one task (e.g., plane) and testing directly on a different task (e.g., car) without any calibration data. Here ELIPformer achieved 8–12 % absolute accuracy gains over the best existing zero‑calibration baselines. The improvement was consistent across all three transfer directions (plane→car, plane→people, car→people), demonstrating that the language‑image prior provides robust, task‑agnostic semantic guidance that bridges the gap between differing visual contexts.

Ablation studies showed that removing either the prompt encoder or the bi‑attention module reduced performance substantially, highlighting the necessity of both components. Moreover, the model retained high performance even when the stimulus images for the new task were unseen during training, confirming the zero‑shot capability inherited from CLIP.

The paper also discusses limitations. CLIP’s pre‑training data are predominantly natural images and English captions; thus, domain‑specific applications (e.g., medical imaging, non‑English prompts) may experience reduced transferability. Computationally, the Transformer‑based EEG encoder and the cross‑attention layers increase inference latency, which could be a bottleneck for real‑time BCI applications. Future work is suggested to explore domain‑adapted language‑image models, lightweight attention mechanisms, and online adaptation strategies.

In summary, this work introduces a pioneering multimodal fusion framework that leverages language‑image pre‑training to supply semantic priors for EEG decoding. By integrating CLIP‑derived embeddings with EEG via bidirectional attention, ELIPformer substantially mitigates the cross‑task zero‑calibration problem that has long hindered the scalability of RSVP‑BCI systems. The publicly released dataset, code, and thorough experimental validation provide a solid foundation for subsequent research aiming to bring BCI technology from controlled laboratory settings to diverse, real‑world environments.