Revisiting Emotions Representation for Recognition in the Wild

Facial emotion recognition has been typically cast as a single-label classification problem of one out of six prototypical emotions. However, that is an oversimplification that is unsuitable for representing the multifaceted spectrum of spontaneous emotional states, which are most often the result of a combination of multiple emotions contributing at different intensities. Building on this, a promising direction that was explored recently is to cast emotion recognition as a distribution learning problem. Still, such approaches are limited in that research datasets are typically annotated with a single emotion class. In this paper, we contribute a novel approach to describe complex emotional states as probability distributions over a set of emotion classes. To do so, we propose a solution to automatically re-label existing datasets by exploiting the result of a study in which a large set of both basic and compound emotions is mapped to probability distributions in the Valence-Arousal-Dominance (VAD) space. In this way, given a face image annotated with VAD values, we can estimate the likelihood of it belonging to each of the distributions, so that emotional states can be described as a mixture of emotions, enriching their description, while also accounting for the ambiguous nature of their perception. In a preliminary set of experiments, we illustrate the advantages of this solution and a new possible direction of investigation. Data annotations are available at https://github.com/jbcnrlz/affectnet-b-annotation.

💡 Research Summary

The paper “Revisiting Emotions Representation for Recognition in the Wild” proposes a fundamentally new paradigm for facial emotion recognition (FER) that moves away from the traditional single‑label classification of six basic emotions. Instead, it treats each face image as a point in the three‑dimensional Valence‑Arousal‑Dominance (VAD) space and converts that point into a probability distribution over a set of emotion classes.

Motivation and Background
Standard FER pipelines either assign one of Ekman’s six basic emotions or predict continuous valence and arousal values. Both approaches ignore the fact that real emotional states are often mixtures of several basic emotions with varying intensities. Recent work on label‑distribution learning (LDL) acknowledges annotation uncertainty but still limits itself to the six basic categories. Moreover, VAD provides a continuous description of affect but lacks a direct mapping to human‑readable emotion terms.

Core Contributions

1. Emotion‑VAD Mapping – The authors adopt Russell et al.’s mapping of 151 emotion terms to 3‑D normal distributions in VAD space. Each emotion E is represented by a mean vector µ_E = (µ_V, µ_A, µ_D) and a diagonal covariance σ_E.
2. Dominance Estimation (CWDE) – Most public datasets contain only valence and arousal. To recover the missing dominance dimension, the paper introduces a Combined Weighted Dominance Estimation (CWDE) method. For each basic emotion a linear regression D = β_0 + β_1 V + β_2 A is fitted using the known VAD statistics. The posterior probability that a given (V,A) pair belongs to each basic emotion provides weights w_i, and the final estimated dominance is a weighted sum of the regression outputs.
3. Emotion Term Selection and Fusion – Using a KD‑tree to find the five nearest neighbors for each emotion, the authors compute the intersection volume of two 3‑D Gaussians via Monte‑Carlo sampling. The Normalized Intersection Measure (NIM) quantifies overlap; if NIM > 0.5 the two emotions are merged into a single prototype. This iterative merging yields a compact taxonomy (typically 10–30 emotions) while preserving the six universal emotions and neutral.
4. Likelihood‑Based Soft Label Generation – Given an image’s VAD coordinates, the likelihood of belonging to each of the K fused emotions is calculated as the product of three univariate Gaussian PDFs (one per V, A, D). Log‑likelihoods are summed, normalized with the LogSumExp trick, and exponentiated to obtain a soft label vector p =