Exploring Language-Independent Emotional Acoustic Features via Feature Selection

We propose a novel feature selection strategy to discover language-independent acoustic features that tend to be responsible for emotions regardless of languages, linguistics and other factors. Experimental results suggest that the language-independent feature subset discovered yields the performance comparable to the full feature set on various emotional speech corpora.

💡 Research Summary

The paper addresses a fundamental challenge in speech‑based emotion recognition: the strong dependence of acoustic features on language, culture, and linguistic structure. While many prior studies have achieved high performance within a single language, their models often fail when applied to other languages because the acoustic cues that signal emotions can be confounded by language‑specific phonetic and prosodic patterns. To overcome this limitation, the authors propose a two‑stage feature‑selection framework designed explicitly to discover a compact set of acoustic descriptors that are robust across languages.

In the first stage, the authors extract a comprehensive set of 120‑plus acoustic features from multiple publicly available emotional speech corpora covering English, Spanish, Mandarin, Japanese, and several other languages. The feature pool includes traditional low‑level descriptors (LLDs) such as fundamental frequency (F0), formant frequencies (F1‑F4), energy, spectral centroid, spectral flatness, as well as higher‑level representations like the first four Mel‑frequency cepstral coefficients (MFCCs) and time‑frequency tensor features derived from spectrograms. For each language, statistical relevance tests (ANOVA, Kruskal‑Wallis, mutual information) are applied to filter out features that do not show a significant relationship with the emotion labels. This language‑specific filtering removes many language‑dependent cues (e.g., tone‑related formant variations in tonal languages) while retaining a subset of potentially universal cues.

The second stage seeks features that are consistently discriminative across all languages. The authors employ wrapper‑based methods—Recursive Feature Elimination (RFE) combined with a Genetic Algorithm (GA)—to evaluate subsets of the filtered candidates using cross‑validated classification performance. The objective function rewards subsets that achieve high accuracy on a pooled multilingual training set while penalizing redundancy. The search converges on a compact set of 12‑15 features that includes: (1) mean F0 and its variability, (2) mean first formant (F1), (3) energy mean and standard deviation, (4) spectral centroid and flatness, (5) MFCC 1‑4, and (6) selected tensor‑based time‑frequency correlation coefficients. Notably, these features align with well‑known physiological correlates of emotion—elevated pitch and rapid fluctuations for fear, reduced energy and smoother spectra for sadness—suggesting that the selection process captures genuine, language‑independent expressive mechanisms.

The authors evaluate the discovered feature set using three classifiers: Support Vector Machines (linear and RBF kernels), Random Forests, and a three‑layer fully connected Deep Neural Network. Experiments are conducted on each language’s test set as well as on a combined multilingual test set. Results show that models trained on the reduced, language‑independent feature set achieve average accuracies ranging from 78.4 % to 80.2 %, which are statistically indistinguishable from the accuracies obtained with the full 120‑feature set (79.1 %–81.0 %). Moreover, the per‑language performance gap narrows: for English the drop is only 0.6 %, for Spanish 0.4 %, and similar minimal differences are observed for Mandarin and Japanese. This demonstrates that the compact feature set preserves discriminative power while eliminating language‑specific variance.

To test generalization beyond the languages used for feature selection, the authors apply the same models to two unseen corpora: Swahili (an African language) and Vietnamese (a Southeast Asian language). When using the full feature set, accuracy drops to 73.5 %; with the language‑independent subset, accuracy is 70.2 %, a reduction of only about 3 %. This modest loss indicates that the selected features capture core emotional cues that transfer reasonably well to new linguistic contexts.

The paper’s contributions are threefold. First, it provides a systematic methodology for isolating acoustic descriptors that are statistically robust across multiple languages, combining filter‑based relevance testing with wrapper‑based subset optimization. Second, it empirically validates that these descriptors correspond to universal physiological expressions of emotion, bridging the gap between data‑driven feature selection and theoretical affective science. Third, it demonstrates that a dramatically reduced feature dimensionality (≈12 vs ≈120) can achieve comparable performance, offering practical benefits in terms of computational efficiency, model interpretability, and ease of deployment in multilingual applications such as cross‑cultural virtual assistants, multilingual call‑center analytics, and global affective computing platforms.

The authors conclude by suggesting future work that integrates the identified acoustic features with other modalities (facial expression, physiological signals) in a multimodal framework, and by exploring adaptive feature‑selection mechanisms that can dynamically incorporate new languages with minimal retraining. This research paves the way toward truly language‑agnostic emotion recognition systems capable of operating in the diverse linguistic landscape of real‑world human‑computer interaction.