Enhancing Personality Recognition by Comparing the Predictive Power of Traits, Facets, and Nuances

Personality is a complex, hierarchical construct typically assessed through item-level questionnaires aggregated into broad trait scores. Personality recognition models aim to infer personality traits from different sources of behavioral data. However, reliance on broad trait scores as ground truth, combined with limited training data, poses challenges for generalization, as similar trait scores can manifest through diverse, context dependent behaviors. In this work, we explore the predictive impact of the more granular hierarchical levels of the Big-Five Personality Model, facets and nuances, to enhance personality recognition from audiovisual interaction data. Using the UDIVA v0.5 dataset, we trained a transformer-based model including cross-modal (audiovisual) and cross-subject (dyad-aware) attention mechanisms. Results show that nuance-level models consistently outperform facet and trait-level models, reducing mean squared error by up to 74% across interaction scenarios.

💡 Research Summary

The paper investigates whether leveraging the hierarchical structure of the Big‑Five personality model—specifically, traits, facets, and the most granular level of individual questionnaire items (referred to as “nuances”)—can improve automatic personality recognition from audiovisual interaction data. The authors argue that using only broad trait scores as ground‑truth labels limits model generalization because individuals with similar trait scores can display highly diverse, context‑dependent behaviors, especially when training data are scarce.

To test this hypothesis, the study employs the UDIVA v0.5 dataset, which contains 80 hours of synchronized video and audio recordings of 134 participants engaged in 145 dyadic sessions. Each participant completed the BFI‑2 questionnaire, yielding 5 trait scores, 15 facet scores (four items per facet), and 60 nuance scores (one item per nuance). The interaction tasks include a free‑talk conversation and three structured games (Ghost, Lego, Animals), providing a variety of social contexts.

Feature extraction follows a spectral approach: visual cues (35 facial action units, gaze vectors, head pose) are obtained via OpenFace 2.0, and the audio signal is processed similarly. All temporal streams are transformed with the Discrete Fourier Transform into fixed‑size 80‑dimensional spectral maps, normalizing sequence length while preserving multi‑scale temporal patterns. These modality‑specific maps are fed into a multimodal transformer architecture based on MulT. The network incorporates intra‑modal self‑attention, cross‑modal attention to fuse complementary cues, and a cross‑subject attention layer that explicitly models the interlocutor’s influence on the target’s behavior.

Training uses a 10‑fold subject‑independent split (≈80 % train, 10 % validation, 10 % test) with mean‑squared error as the loss function, Adam optimizer, and Bayesian hyper‑parameter tuning for learning rate and batch size. Separate models are trained for each hierarchy level (traits, facets, nuances) and for each interaction task, resulting in a total of 12 primary models. Performance is evaluated with four metrics: MSE, MAE, Pearson correlation coefficient (PCC), and R². Baselines are simple averages of training labels applied to the test set.

Results consistently show that nuance‑level models outperform both facet‑level and trait‑level models across all tasks and metrics. For example, the average MSE for the nuance model on the “Talk” task is 0.0492, compared with 0.1741 for the trait model (≈71 % reduction) and 0.1185 for the facet model (≈58 % reduction). Similar gains are observed for MAE, PCC (up to 0.95 for nuances versus ~0.71 for traits), and R² (up to 0.83 versus ~0.34). Facet models also improve over traits, though the margin is smaller. The most challenging trait—Negative Emotionality—benefits dramatically from nuance modeling, achieving an MSE of 0.0256 (the lowest among all configurations).

Task‑level analysis reveals negligible performance differences across the four interaction scenarios, suggesting that the spectral representation and cross‑subject attention effectively capture context‑invariant personality cues. Additional aggregation experiments (median instead of mean, aggregating before conversion to traits) resulted in lower performance, indicating that averaging nuances into facets and then into traits preserves useful signal while reducing noise.

The authors acknowledge limitations: the high dimensionality of nuance labels can exacerbate data sparsity, and averaging predictions across multiple sessions may dilute fine‑grained behavioral information. Future work is proposed to explore session‑wise weighting, multi‑task learning that jointly predicts traits, facets, and nuances, integration of textual or linguistic cues, and real‑time personality inference for adaptive human‑machine interaction.

Overall, the study provides strong empirical evidence that moving down the personality hierarchy to the item level yields substantially more accurate personality recognition from multimodal behavioral data, opening new avenues for fine‑grained affective computing and personalized AI systems.