Likelihood-based semi-supervised model selection with applications to speech processing

In conventional supervised pattern recognition tasks, model selection is typically accomplished by minimizing the classification error rate on a set of so-called development data, subject to ground-truth labeling by human experts or some other means. In the context of speech processing systems and other large-scale practical applications, however, such labeled development data are typically costly and difficult to obtain. This article proposes an alternative semi-supervised framework for likelihood-based model selection that leverages unlabeled data by using trained classifiers representing each model to automatically generate putative labels. The errors that result from this automatic labeling are shown to be amenable to results from robust statistics, which in turn provide for minimax-optimal censored likelihood ratio tests that recover the nonparametric sign test as a limiting case. This approach is then validated experimentally using a state-of-the-art automatic speech recognition system to select between candidate word pronunciations using unlabeled speech data that only potentially contain instances of the words under test. Results provide supporting evidence for the utility of this approach, and suggest that it may also find use in other applications of machine learning.

💡 Research Summary

The paper addresses a fundamental bottleneck in large‑scale pattern‑recognition systems: the need for labeled development data when selecting among competing statistical models. In conventional supervised settings, model selection is performed by minimizing classification error on a development set that has been manually annotated. However, in domains such as speech processing, obtaining reliable ground‑truth labels is costly, time‑consuming, and often impractical at the scale required for modern systems.

To overcome this limitation, the authors propose a semi‑supervised, likelihood‑based framework that exploits unlabeled data. The key idea is to train a separate classifier for each candidate model (e.g., a language model representing a particular pronunciation variant). These classifiers are then used to automatically generate “putative” labels for the unlabeled speech corpus. Because the automatically generated labels are inevitably noisy, the framework incorporates robust‑statistics concepts to handle labeling errors in a principled way.

The theoretical contribution centers on a censored likelihood‑ratio test. For each observation, the test either uses the full likelihood ratio or discards (censors) the observation if its contribution falls below a pre‑specified threshold. By optimally choosing the censoring level, the test attains minimax‑optimality under a bounded contamination model: the worst‑case error probability is minimized over all possible labeling error rates up to a prescribed bound. Remarkably, when the censoring threshold approaches zero, the test reduces to the non‑parametric sign test, establishing a direct link between robust likelihood‑ratio testing and a classic distribution‑free procedure.

The authors validate the approach on a state‑of‑the‑art automatic speech recognition (ASR) system. The experimental scenario involves selecting the most appropriate pronunciation for a set of target words. Unlabeled speech recordings—some of which contain the target words, others do not—are processed by each candidate pronunciation model’s decoder, producing hypothesized word sequences that serve as putative labels. The censored likelihood‑ratio test is then applied to compare the models.

Results demonstrate three important findings. First, the semi‑supervised method achieves recognition performance comparable to, and in some cases exceeding, that of fully supervised cross‑validation, despite using virtually no human‑annotated data. Second, the method is robust to labeling noise: as long as the contamination proportion stays below the designed bound, the sign‑test‑derived decision rule remains near‑optimal. Third, the censoring parameter provides a practical knob for trading off sensitivity to labeling errors against statistical power, allowing system designers to adapt the procedure to varying noise conditions.

Beyond speech processing, the paper’s framework offers a general recipe for likelihood‑based model selection when labeled data are scarce. By coupling model‑specific classifiers with robust, censored likelihood testing, practitioners can obtain minimax‑optimal decisions without incurring the prohibitive cost of manual annotation. The authors suggest future extensions to multi‑class problems, continuous‑label contamination models, and applications in computer vision and natural‑language processing, where similar labeling bottlenecks exist. Overall, the work bridges a gap between theoretical robust statistics and practical semi‑supervised learning, providing a scalable solution for model selection in real‑world machine‑learning pipelines.