Decoding Speech Envelopes from Electroencephalogram with a Contrastive Pearson Correlation Coefficient Loss

Recent advances in reconstructing speech envelopes from Electroencephalogram (EEG) signals have enabled continuous auditory attention decoding (AAD) in multi-speaker environments. Most Deep Neural Network (DNN)-based envelope reconstruction models are trained to maximize the Pearson correlation coefficients (PCC) between the attended envelope and the reconstructed envelope (attended PCC). While the difference between the attended PCC and the unattended PCC plays an essential role in auditory attention decoding, existing methods often focus on maximizing the attended PCC. We therefore propose a contrastive PCC loss which represents the difference between the attended PCC and the unattended PCC. The proposed approach is evaluated on three public EEG AAD datasets using four DNN architectures. Across many settings, the proposed objective improves envelope separability and AAD accuracy, while also revealing dataset- and architecture-dependent failure cases.

💡 Research Summary

This paper addresses a fundamental limitation in EEG‑based auditory attention decoding (AAD) that arises from the way speech‑envelope reconstruction models are trained. Most deep‑neural‑network (DNN) approaches optimize a simple Pearson‑correlation loss (L_PCC = −ρ_a), which only maximizes the correlation between the reconstructed envelope and the attended speech envelope. While this improves the attended Pearson correlation coefficient (PCC), it does not explicitly discourage the model from also correlating with unattended speech, thereby limiting the contrast between attended and unattended PCCs—a contrast that is essential for reliable AAD.

To remedy this, the authors propose a contrastive PCC loss (L_ΔPCC) that simultaneously maximizes the attended PCC and minimizes the mean unattended PCC:
L_ΔPCC = −ρ_a + (1/(N‑1)) ∑{j=1}^{N‑1} ρ{u,j}.
Here ρ_a is the Pearson correlation between the predicted envelope and the attended speech, ρ_{u,j} are the correlations with each of the N‑1 unattended speakers, and N is the total number of concurrent speakers. By using the average of unattended PCCs rather than the sum, the loss avoids a pathological solution in which the model drives all correlations negative to reduce the loss without learning meaningful representations.

The authors evaluate this loss on three publicly available two‑talker EEG datasets—KUL, DTU, and KUL‑AV‑GC—each recorded with a 64‑channel BioSemi system, band‑pass filtered (1–32 Hz), and down‑sampled to 128 Hz. Speech envelopes are extracted using a 17‑band ERB filterbank, power‑law compressed (exponent 0.6), and summed to a broadband envelope. Four state‑of‑the‑art regression architectures are tested: (1) VLAAI (stacked 1‑D convolutions with skip connections), (2) LSM‑CNN (learnable spatial‑mapping layer converting 2‑D EEG to a 3‑D layout), (3) EEG‑Mamba (state‑space Mamba blocks with multi‑head self‑attention for mel‑spectrogram reconstruction), and (4) EEG‑Deformer (hybrid CNN‑Transformer). Each model is trained separately with L_PCC and L_ΔPCC using a four‑fold leave‑one‑trial‑out cross‑validation, AdamW optimizer (lr = 5 × 10⁻⁴, weight decay = 5 × 10⁻⁴), batch size 64, up to 100 epochs with early stopping.

Performance is measured by (i) decoding accuracy—the proportion of trials where attended PCC exceeds all unattended PCCs—and (ii) ΔPCC, the numerical difference between attended and mean unattended PCCs. Across all combinations, models trained with L_ΔPCC generally achieve higher decoding accuracy and larger ΔPCC values. On average, decoding accuracy improves by 2–4 percentage points, while ΔPCC increases by 17.8 % relative to the baseline loss. Linear regression analysis shows a moderate‑to‑strong correlation (R² > 0.5) between ΔPCC and decoding accuracy, whereas the correlation with attended PCC alone is weaker. The benefit is most pronounced for longer analysis windows (e.g., 10 s segments) and for the VLAAI and LSM‑CNN architectures, which display the steepest slopes in the ΔPCC‑vs‑accuracy plots.

Nevertheless, the improvement is not universal. EEG‑Mamba on the DTU dataset shows a slight drop in accuracy when trained with L_ΔPCC, and some model‑dataset pairs exhibit negligible gains despite a substantial ΔPCC increase. The authors attribute these inconsistencies to (a) window‑length effects—short windows provide insufficient signal‑to‑noise ratio for the contrastive objective to be effective, and (b) dataset heterogeneity—differences in language, stimulus presentation, and head‑related impulse responses introduce variability that can interact unfavorably with the loss formulation.

The paper’s contributions are threefold: (1) introduction of a contrastive Pearson‑correlation loss that explicitly enhances the attended‑unattended envelope separation, (2) systematic benchmarking of this loss across multiple DNN architectures and three benchmark EEG datasets, and (3) empirical evidence that ΔPCC is a more reliable predictor of AAD performance than attended PCC alone. The authors conclude that while contrastive loss offers a valuable tool for improving EEG‑based speech‑envelope reconstruction, its efficacy depends on data characteristics, model capacity, and analysis window size. Future work should explore adaptive weighting of attended vs. unattended terms, integration with end‑to‑end attention classifiers, and robustness across diverse acoustic‑neural environments.