Research on several key technologies in practical speech emotion recognition

In this dissertation the practical speech emotion recognition technology is studied, including several cognitive related emotion types, namely fidgetiness, confidence and tiredness. The high quality of naturalistic emotional speech data is the basis of this research. The following techniques are used for inducing practical emotional speech: cognitive task, computer game, noise stimulation, sleep deprivation and movie clips. A practical speech emotion recognition system is studied based on Gaussian mixture model. A two-class classifier set is adopted for performance improvement under the small sample case. Considering the context information in continuous emotional speech, a Gaussian mixture model embedded with Markov networks is proposed. A further study is carried out for system robustness analysis. First, noise reduction algorithm based on auditory masking properties is fist introduced to the practical speech emotion recognition. Second, to deal with the complicated unknown emotion types under real situation, an emotion recognition method with rejection ability is proposed, which enhanced the system compatibility against unknown emotion samples. Third, coping with the difficulties brought by a large number of unknown speakers, an emotional feature normalization method based on speaker-sensitive feature clustering is proposed. Fourth, by adding the electrocardiogram channel, a bi-modal emotion recognition system based on speech signals and electrocardiogram signals is first introduced. The speech emotion recognition methods studied in this dissertation may be extended into the cross-language speech emotion recognition and the whispered speech emotion recognition.

💡 Research Summary

**
This dissertation tackles the problem of practical speech‑based emotion recognition by focusing on three cognitively relevant affective states—fidgetiness (anxiety), confidence, and tiredness. The work begins with the construction of a high‑quality, naturalistic emotional speech corpus. Five emotion‑induction protocols are designed: cognitive tasks, computer games, noise stimulation, sleep deprivation, and movie clips. Each protocol reliably elicits one of the target emotions, and recordings are collected from a diverse pool of speakers (varying gender, age, and cultural background). Simultaneously, electrocardiogram (ECG) signals are captured to enable multimodal analysis.

The core recognition engine is built on Gaussian Mixture Models (GMMs). To mitigate the well‑known small‑sample problem, the author introduces a two‑class classifier set: for every pair of emotions a dedicated binary GMM classifier is trained, and the final decision is obtained by aggregating the binary outcomes. This decomposition reduces model complexity and prevents over‑fitting when training data are scarce. Recognizing that emotional speech is continuous, a Markov network is embedded within the GMM framework. The Markov layer models temporal state transitions between adjacent frames, allowing the system to smooth isolated misclassifications by leveraging contextual information.

Robustness is addressed through four complementary techniques. First, a noise‑reduction front‑end exploits auditory masking properties: frequency‑dependent masking thresholds are estimated and used to suppress noise components that fall below the human hearing mask, preserving salient emotional cues even at low signal‑to‑noise ratios. Second, an emotion‑rejection mechanism is implemented. If the posterior probability of any known class falls below a pre‑defined threshold, the sample is labeled as “unknown,” thereby protecting the system from forced misclassification of novel or ambiguous emotions. Third, speaker variability is handled by speaker‑sensitive feature clustering. All speaker data are clustered, and each cluster’s mean and variance are used to normalize acoustic features, which substantially reduces inter‑speaker variance. Fourth, a bimodal architecture combines speech with ECG. Feature‑level fusion concatenates acoustic and cardiac descriptors, while decision‑level fusion averages the posterior probabilities from the two modalities. Experiments show that ECG contributes most to distinguishing fatigue, where physiological changes are more pronounced than acoustic ones.

Extensive experiments validate each component. The GMM‑Markov model outperforms a plain GMM by an average of 6.3 percentage points on continuous emotional speech. In noisy conditions, the masking‑based denoiser maintains performance with only a 2 percentage‑point drop when SNR is reduced by 5 dB. The rejection scheme slightly lowers overall accuracy (≈1 pp) but cuts the misclassification rate of unknown emotions by over 40 %. Speaker‑sensitive normalization yields a 4.1 percentage‑point gain on a test set with high speaker diversity. Finally, the speech‑ECG bimodal system improves fatigue recognition by more than 8 percentage points compared with speech‑only baselines.

The dissertation also discusses scalability. The proposed methods can be transferred to cross‑language emotion recognition by initializing GMM parameters with language‑specific priors and adapting the Markov transition matrix to language‑specific prosodic patterns. For whispered speech, where acoustic energy is low, the ECG channel provides complementary physiological cues, preserving recognition capability. Limitations include the focus on only three emotions, reliance on laboratory‑controlled induction scenarios, and the need for real‑world, long‑duration deployments. Future work should expand the emotion taxonomy, collect in‑the‑wild data, and explore real‑time implementation on embedded platforms. Overall, the study presents a comprehensive, technically sound framework for building robust, practical speech emotion recognition systems that can operate under noisy, speaker‑variable, and multimodal conditions.