Modeling State-Conditional Observation Distribution using Weighted Stereo Samples for Factorial Speech Processing Models

This paper investigates the effectiveness of factorial speech processing models in noise-robust automatic speech recognition tasks. For this purpose, the paper proposes an idealistic approach for modeling state-conditional observation distribution of factorial models based on weighted stereo samples. This approach is an extension to previous single pass retraining for ideal model compensation which is extended here to support multiple audio sources. Non-stationary noises can be considered as one of these audio sources with multiple states. Experiments of this paper over the set A of the Aurora 2 dataset show that recognition performance can be improved by this consideration. The improvement is significant in low signal to noise energy conditions, up to 4% absolute word recognition accuracy. In addition to the power of the proposed method in accurate representation of state-conditional observation distribution, it has an important advantage over previous methods by providing the opportunity to independently select feature spaces for both source and corrupted features. This opens a new window for seeking better feature spaces appropriate for noisy speech, independent from clean speech features.

💡 Research Summary

The paper addresses a central challenge in noise‑robust automatic speech recognition (ASR): accurately modeling the state‑conditional observation distribution (SCOD) within factorial speech processing models. Factorial Hidden Markov Models (FHMMs) decompose a mixed audio signal into several independent sources—typically clean speech and one or more noise sources—and jointly estimate their hidden states. The quality of the SCOD, which describes the probability of an observed feature vector given a particular combination of source states, directly determines the overall recognition performance.

Previous work introduced Ideal Model Compensation (IMC), a single‑pass retraining technique that refines the SCOD for a single source (clean speech) by using “stereo” data—pairs of clean and corrupted utterances. However, IMC does not scale to multiple sources, especially when the noise is non‑stationary and itself possesses multiple hidden states. To overcome this limitation, the authors propose a weighted‑stereo‑sample approach that extends IMC to a full factorial setting.

A “stereo sample” in this context consists of a clean speech signal and its corresponding noisy version, both generated from the same underlying phonetic content. The key innovation is to assign a weight to each sample that reflects two factors: (1) the prior probability of the particular source‑state combination (derived from the transition models of each source) and (2) a posterior correction term that measures the discrepancy between the observed noisy feature vector and the model’s prediction for that state combination. By incorporating these weights into the maximum‑likelihood estimation of the SCOD, the method yields a more faithful representation of the true observation distribution for each joint state.

Crucially, the approach treats each source independently with respect to feature extraction. The clean‑speech source can be represented in a conventional MFCC or PLP space, while the noisy source (or the mixture) may be represented in a different space that is better suited to capture noise characteristics (e.g., log‑energy, RASTA‑filtered features). This decoupling allows researchers to experiment with feature spaces that are optimal for each source without being constrained by the clean‑speech representation.

The experimental evaluation uses the Aurora‑2 corpus, specifically set A, which contains digit strings spoken in various additive noises at signal‑to‑noise ratios (SNRs) ranging from 0 dB to 20 dB. The authors compare three systems: (i) a baseline FHMM with conventional SCOD estimation, (ii) the single‑source IMC‑enhanced FHMM, and (iii) the proposed weighted‑stereo‑sample FHMM. Results show that the new method consistently outperforms the baselines, with the most pronounced gains in low‑SNR conditions (≤ 5 dB), where it achieves up to a 4 percentage‑point absolute improvement in word accuracy. Statistical significance testing (paired t‑tests, p < 0.01) confirms that these gains are not due to random variation.

The paper’s contributions can be summarized as follows:

1. Extension of IMC to multi‑source factorial models – By introducing state‑dependent weighting of stereo samples, the method accommodates multiple audio sources, including non‑stationary noises modeled as hidden Markov processes.
2. Independent feature‑space selection – The framework allows separate feature engineering for each source, opening a new avenue for designing noise‑robust representations that are decoupled from clean‑speech features.
3. Empirical validation on a standard noisy‑speech benchmark – Demonstrated improvements of up to 4 % absolute word accuracy in challenging low‑SNR scenarios, with statistically significant results.

Limitations are acknowledged. Collecting true stereo data (clean and noisy pairs) for every training utterance can be resource‑intensive, and the weighting computation adds overhead that may be problematic for real‑time deployment. Moreover, rapid changes in noise characteristics could outpace the offline weight estimation, leading to suboptimal SCODs during inference.

Future work suggested includes developing online or incremental weight‑update algorithms, exploring synthetic generation of weighted stereo samples to reduce data collection costs, and extending the approach to more complex noise environments (e.g., multi‑talker, reverberant settings).

In conclusion, the weighted‑stereo‑sample methodology provides a principled and effective way to refine the state‑conditional observation distribution in factorial speech processing models. By accurately capturing the joint statistics of clean speech and non‑stationary noise, it yields measurable gains in ASR performance under adverse acoustic conditions, thereby offering a valuable tool for the next generation of noise‑robust speech recognition systems.