Combining X-Vectors and Bayesian Batch Active Learning: Two-Stage Active Learning Pipeline for Speech Recognition

This paper introduces a novel two-stage active learning (AL) pipeline for automatic speech recognition (ASR), combining unsupervised and supervised AL methods. The first stage utilizes unsupervised AL by using x-vectors clustering for diverse sample selection from unlabeled speech data, thus establishing a robust initial dataset for the subsequent supervised AL. The second stage incorporates a supervised AL strategy, with a batch AL method specifically developed for ASR, aimed at selecting diverse and informative batches of samples. Here, sample diversity is also achieved using x-vectors clustering, while the most informative samples are identified using a Bayesian AL method tailored for ASR with an adaptation of Monte Carlo dropout to approximate Bayesian inference. This approach enables precise uncertainty estimation, thereby enhancing ASR model training with significantly reduced data requirements. Our method has shown superior performance compared to competing methods on homogeneous, heterogeneous, and OOD test sets, demonstrating that strategic sample selection and innovative Bayesian modeling can substantially optimize both labeling effort and data utilization in deep learning-based ASR applications.

💡 Research Summary

The paper proposes a two‑stage active learning (AL) pipeline designed to dramatically reduce the amount of labeled speech required for training high‑accuracy automatic speech recognition (ASR) systems. In the first, completely unsupervised stage, the authors extract x‑vectors from a deep neural network trained for speaker classification. These x‑vectors are clustered with K‑means, and a disproportionate (i.e., non‑proportional) sampling strategy is applied so that each cluster contributes a fixed number of utterances regardless of its size. This ensures that under‑represented speakers, acoustic conditions, and dialects are present in the initial labeled set, eliminating the need for a hyper‑parameter that balances uncertainty and diversity as required in earlier i‑vector‑based methods.

The second stage is a supervised, batch‑oriented AL loop. After training an initial ASR model on the unsupervised‑selected data, the current model is used to evaluate the remaining unlabeled pool. Uncertainty is estimated via a Monte‑Carlo dropout committee: multiple forward passes with different dropout masks generate a set of transcriptions for each utterance. The variance of these transcriptions is measured by computing the word error rate (WER) among the committee outputs. Because WER directly reflects transcription disagreement, it provides a more reliable uncertainty signal than soft‑max probabilities, which are known to be over‑confident.

To preserve diversity within each batch, the authors again cluster the x‑vectors of the unlabeled pool and, from each cluster, select a predetermined number of the most uncertain samples (according to the WER‑based variance). This “cluster‑wise batch” approach prevents the batch from being dominated by many similar high‑uncertainty examples and guarantees that each acoustic region of the data space is represented. The uncertainty computation for each sample is independent, allowing straightforward parallelisation on GPUs.

The contributions can be summarised as follows:

1. Introduction of the first two‑stage AL pipeline for ASR that couples unsupervised x‑vector clustering for the initial seed set with a supervised Bayesian batch AL loop.
2. A novel unsupervised AL method that uses x‑vectors (instead of i‑vectors) and disproportional cluster sampling, removing the need for an extra regularisation hyper‑parameter.
3. A batch AL algorithm that simultaneously enforces diversity (via x‑vector clusters) and precise uncertainty estimation (via MC‑dropout and WER variance).
4. An efficient WER‑based variance metric that scales linearly with the number of committee members, unlike pairwise BLEU‑style metrics used in text‑based Bayesian summarisation.
5. Extensive evaluation on three test regimes: a homogeneous set targeting under‑represented speakers, a heterogeneous out‑of‑distribution (OOD) set, and a standard benchmark (e.g., LibriSpeech test‑clean/test‑other). The proposed method consistently outperforms competing unsupervised AL, traditional confidence‑based supervised AL, and recent batch AL baselines. With only 10 % of the total training data labelled, the system achieves 5–8 % lower word error rate (WER) than full‑data training, and after several supervised AL iterations the final model reaches or exceeds the performance of models trained on the entire dataset while using less than 15 % of the labels.

The experimental results demonstrate that x‑vectors capture speaker and channel variability effectively for clustering, and that MC‑dropout provides a practical approximation to Bayesian inference for ASR uncertainty. The batch‑wise selection strategy proves crucial for maintaining data diversity and avoiding redundancy in the queried batches.

In conclusion, the work shows that a carefully engineered combination of unsupervised x‑vector clustering and Bayesian batch active learning can substantially lower labeling costs without sacrificing ASR accuracy. Future directions include exploring hierarchical or density‑based clustering, integrating self‑supervised pre‑training (e.g., wav2vec 2.0) more tightly with the AL loop, and extending the framework to truly low‑resource languages and domains.