What Does the Speaker Embedding Encode?

Developing a good speaker embedding has received tremendous interest in the speech community, with representations such as i-vector and d-vector demonstrating remarkable performance across various tasks. Despite their widespread adoption, a fundamental question remains largely unexplored: what properties are actually encoded in these embeddings? To address this gap, we conduct a comprehensive analysis of three prominent speaker embedding methods: i-vector, d-vector, and RNN/LSTM-based sequence-vector (s-vector). Through carefully designed classification tasks, we systematically investigate their encoding capabilities across multiple dimensions, including speaker identity, gender, speaking rate, text content, word order, and channel information. Our analysis reveals distinct strengths and limitations of each embedding type: i-vector excels at speaker discrimination but encodes limited sequential information; s-vector captures text content and word order effectively but struggles with speaker identity; d-vector shows balanced performance but loses sequential information through averaging. Based on these insights, we propose a novel multi-task learning framework that integrates i-vector and s-vector, resulting in a new speaker embedding (i-s-vector) that combines their complementary advantages. Experimental results on RSR2015 demonstrate that the proposed i-s-vector achieves more than 50% EER reduction compared to the i-vector baseline on content mismatch trials, validating the effectiveness of our approach.

💡 Research Summary

The paper investigates what information is actually encoded in three widely used speaker embedding methods—i‑vector, d‑vector, and s‑vector—by designing a suite of classification tasks that probe various properties of speech utterances. The authors first review the evolution of speaker representations, from GMM‑UBM and super‑vectors to the total‑variability i‑vector, deep‑learning based d‑vector, and recurrent‑network based s‑vector. They then propose a systematic analysis methodology: if a property is encoded in an embedding, a simple classifier should be able to predict that property from the embedding with high accuracy. To keep the focus on the embeddings themselves, a single‑hidden‑layer multilayer perceptron (MLP) with ReLU activation is used for all tasks.

Eight prediction tasks are constructed, covering three broad categories:

1. Speaker‑related properties – speaker identity (106 classes), gender (binary), speaking rate (slow/normal/fast).
2. Text‑related properties – spoken sentence (30 classes), spoken term (binary for each of 147 words), word order (binary whether utterance A precedes B), utterance length (four duration bins).
3. Channel‑related property – recording handset (six classes).

All experiments use the RSR2015 Part‑1 dataset. The background subset (97 speakers) trains the embedding extractors, while the evaluation subset (106 speakers) supplies the data for the classification tasks. Feature extraction uses 39‑dimensional PLP (static + delta + delta‑delta). The i‑vector system employs a 1024‑component GMM‑UBM and total‑variability matrix (400‑600 dimensions). The d‑vector is a 5‑layer DNN (1024 units per hidden layer) trained to classify the 97 background speakers; utterance‑level vectors are obtained by averaging the last hidden‑layer activations. The s‑vector uses a unidirectional LSTM; to mitigate the scarcity of utterance‑level training samples, a multitask objective predicts both speaker identity and the spoken sentence (30 classes).

Key findings from the classification results:

• Speaker identity: i‑vector achieves the highest accuracy (≈90 %), confirming its strong capacity to capture speaker characteristics. s‑vector performs poorly (≈60 %) due to limited utterance‑level training data, while d‑vector sits in the middle (≈75 %).
• Speech text (sentence) classification: Both i‑vector and s‑vector reach near‑perfect accuracy (≈100 %). d‑vector also performs well (≈95 %) despite averaging, indicating that the DNN still preserves substantial lexical information.
• Spoken term detection: When the embedding dimension exceeds 300, s‑vector outperforms i‑vector, showing superior word‑level encoding. d‑vector fails almost completely, confirming that averaging discards fine‑grained lexical cues.
• Word order: d‑vector and i‑vector both hover around chance (≈50 %), whereas s‑vector attains ≈98 % accuracy, demonstrating that recurrent models naturally retain sequential order while the other methods do not.
• Utterance length: i‑vector and s‑vector both exceed 70 % accuracy, indicating that duration cues survive in these embeddings; d‑vector lags (≈45 %).
• Channel, gender, speaking rate: All three embeddings predict channel and gender with >70 % accuracy, but speaking rate is less reliably captured (≈60 % for i‑vector and s‑vector, ≈55 % for d‑vector).

From these observations the authors conclude that each embedding type has distinct strengths: i‑vector excels at speaker discrimination but lacks sequential information; s‑vector captures lexical content and order but is weak on speaker identity; d‑vector offers a balanced trade‑off but loses sequential cues due to averaging.

Proposed i‑s‑vector: To combine the complementary advantages, a multi‑task learning framework jointly optimizes i‑vector and s‑vector objectives, and the final embedding is formed by concatenating the two vectors. Experiments on text‑mismatch trials of RSR2015 show that i‑s‑vector reduces equal error rate (EER) by more than 50 % relative to the i‑vector baseline, confirming its robustness to content variation.

Implications: The study provides a practical diagnostic toolkit for understanding what speaker embeddings encode, guiding the selection of representations for specific applications (e.g., text‑dependent verification vs. text‑independent speaker identification). It also demonstrates that hybrid, multitask‑trained embeddings can achieve superior performance by leveraging the strengths of both discriminative (i‑vector) and sequential (s‑vector) modeling. The methodology and findings are likely to influence future research on speaker representation learning, especially in scenarios where both speaker identity and lexical content are important.