Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings

Self-supervised speaker embeddings are widely used in speaker verification systems, but prior work has shown that they often encode sensitive demographic attributes, raising fairness and privacy concerns. This paper investigates the extent to which demographic information, specifically gender, age, and accent, is present in SimCLR-trained speaker embeddings and whether such leakage can be mitigated without severely degrading speaker verification performance. We study two debiasing strategies: adversarial training through gradient reversal and a causal bottleneck architecture that explicitly separates demographic and residual information. Demographic leakage is quantified using both linear and nonlinear probing classifiers, while speaker verification performance is evaluated using ROC-AUC and EER. Our results show that gender information is strongly and linearly encoded in baseline embeddings, whereas age and accent are weaker and primarily nonlinearly represented. Adversarial debiasing reduces gender leakage but has limited effect on age and accent and introduces a clear trade-off with verification accuracy. The causal bottleneck further suppresses demographic information, particularly in the residual representation, but incurs substantial performance degradation. These findings highlight fundamental limitations in mitigating demographic leakage in self-supervised speaker embeddings and clarify the trade-offs inherent in current debiasing approaches.

💡 Research Summary

This paper investigates the presence of demographic information—specifically gender, age, and accent—in self‑supervised speaker embeddings learned with SimCLR, and evaluates two mitigation strategies: adversarial debiasing via gradient reversal and a causal bottleneck architecture that explicitly separates demographic from residual information. Using the Mozilla Common Voice English corpus, the authors curate a balanced subset with binary gender labels, three age groups, and five accent categories, ending up with 11,209 speakers split into training, validation, and test sets. SimCLR is trained to maximize agreement between augmented views of the same utterance, producing embeddings that are highly discriminative for speaker verification (measured by ROC‑AUC and Equal Error Rate).

To quantify leakage, the study employs both linear probes (logistic regression for gender and accent, linear regression for age) and nonlinear probes (three‑layer MLPs). Results show that gender is almost perfectly linearly separable (≈99 % accuracy), indicating a dominant gender direction in the embedding space. Age and accent, however, are encoded more diffusely: linear probes achieve modest performance, while nonlinear probes substantially improve accuracy (age MAE ≈4.5 years, accent ≈78 % classification), revealing that these attributes are spread across many dimensions and require nonlinear mappings to be extracted.

The first mitigation approach adds a gradient‑reversal layer (GRL) and an adversarial classifier that predicts each demographic attribute from the embeddings. During training, the GRL flips the gradient from the adversary, encouraging the encoder to remove demographic cues. This method reduces gender leakage by roughly 30 % but has limited impact on age and accent, especially on their nonlinear components. Moreover, it introduces a clear fairness‑utility trade‑off: speaker verification performance drops (ROC‑AUC decreases by ~1.2 %, EER rises by ~0.4 %).

The second approach introduces a causal bottleneck at the end of the encoder. The bottleneck splits the representation into two sub‑spaces—one dedicated to demographic factors (explicitly constrained) and one for residual speaker‑specific information. By limiting the flow of demographic information, the bottleneck further suppresses leakage: gender probe accuracy falls below 85 %, and age/accent nonlinear probes lose about 10 % of their predictive power. However, the price is steep—verification ROC‑AUC declines by more than 3 % and EER increases by over 1 %, indicating substantial degradation of the primary task. Residual leakage persists in the “speaker” sub‑space, showing that complete disentanglement is difficult.

Overall, the paper demonstrates three key insights: (1) Self‑supervised contrastive learning, while powerful for speaker discrimination, naturally inherits demographic biases present in the training data. (2) Adversarial debiasing can effectively attenuate linearly encoded bias (gender) but struggles with nonlinear bias and incurs modest performance loss. (3) Causal bottleneck architectures achieve stronger bias suppression but at the cost of significant verification accuracy degradation. The authors conclude that future work must explore more sophisticated multi‑objective optimization, information‑theoretic regularization, or meta‑learning strategies to achieve a better balance between fairness (privacy) and utility in multilingual speaker embedding systems.