MS-Mix: Unveiling the Power of Mixup for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) aims to identify and interpret human emotions by integrating information from heterogeneous data sources such as text, video, and audio. While deep learning models have advanced in network architecture design, they remain heavily limited by scarce multimodal annotated data. Although Mixup-based augmentation improves generalization in unimodal tasks, its direct application to MSA introduces critical challenges: random mixing often amplifies label ambiguity and semantic inconsistency due to the lack of emotion-aware mixing mechanisms. To overcome these issues, we propose MS-Mix, an adaptive, emotion-sensitive augmentation framework that automatically optimizes sample mixing in multimodal settings. The key components of MS-Mix include: (1) a Sentiment-Aware Sample Selection (SASS) strategy that effectively prevents semantic confusion caused by mixing samples with contradictory emotions. (2) a Sentiment Intensity Guided (SIG) module using multi-head self-attention to compute modality-specific mixing ratios dynamically based on their respective emotional intensities. (3) a Sentiment Alignment Loss (SAL) that aligns the prediction distributions across modalities, and incorporates the Kullback-Leibler-based loss as an additional regularization term to train the emotion intensity predictor and the backbone network jointly. Extensive experiments on three benchmark datasets with six state-of-the-art backbones confirm that MS-Mix consistently outperforms existing methods, establishing a new standard for robust multimodal sentiment augmentation. The source code is available at: https://github.com/HongyuZhu-s/MS-Mix.

💡 Research Summary

The paper introduces MS‑Mix, an emotion‑aware mixup augmentation framework designed specifically for multimodal sentiment analysis (MSA), where textual, visual, and acoustic streams are jointly processed to infer continuous sentiment intensity scores. The authors first identify a critical shortcoming of conventional mixup methods: random pairing of samples and fixed mixing ratios often produce semantically inconsistent mixtures, especially when opposite emotions are blended, leading to label noise and degraded model performance. To address this, MS‑Mix comprises three tightly integrated components.

1. Sentiment‑Aware Sample Selection (SASS) – After extracting modality‑specific embeddings via a backbone encoder, the method computes an averaged semantic similarity across modalities for each candidate pair. Only pairs whose similarity exceeds a predefined threshold δ are retained, effectively filtering out incompatible samples that would otherwise introduce contradictory emotional cues.

2. Sentiment Intensity Guided (SIG) mixing – A multi‑head self‑attention module predicts an emotion intensity vector for each modality. These intensities condition the computation of modality‑specific mixing coefficients λᵐᵢⱼ, allowing the model to assign higher weight to the modality that carries stronger emotional evidence for a given pair. Consequently, the mixed feature ẑᵐ = λᵐᵢⱼ·zᵐᵢ + (1‑λᵐᵢⱼ)·zᵐⱼ and mixed label ŷ = λᴸᵢⱼ·yᵢ + (1‑λᴸᵢⱼ)·yⱼ are generated in a data‑driven, emotion‑sensitive manner.

3. Sentiment Alignment Loss (SAL) – To ensure that the augmented samples remain faithful to the true sentiment distribution, a KL‑divergence term aligns the predicted sentiment distribution P_L of the mixed sample with the original intensity distribution P_m. This regularizer jointly optimizes the backbone, the intensity predictor, and the mixing module, reducing label smoothing and improving inter‑modal consistency.

The total training objective combines the original regression loss (MSE), the mixed‑sample regression loss, and the SAL term, each weighted by hyper‑parameters ξ₁ and ξ₂.

Experiments are conducted on three widely used benchmarks—CMU‑MOSEI, MOSI, and IEMOCAP—using six state‑of‑the‑art backbones (e.g., BERT for text, Transformer‑based video encoders, CNN‑based audio encoders). Across all settings, MS‑Mix consistently outperforms strong baselines such as Manifold‑Mixup, MultiMix, and PowMix, achieving higher F1 scores, lower mean absolute error (MAE), and reduced KL divergence between generated and true label distributions. Ablation studies demonstrate that removing any of SASS, SIG, or SAL leads to noticeable performance drops, confirming the necessity of each component. Visualizations of the selected pairs and mixing ratios illustrate that SASS indeed picks semantically aligned samples, while SIG dynamically adjusts modality contributions based on emotion intensity.

The authors acknowledge limitations: the quality of the intensity predictor directly influences augmentation effectiveness, and the additional attention‑based computations increase training overhead. Future work is suggested on lightweight intensity estimators, unsupervised similarity metrics, and extending the framework to noisy, real‑world streams such as social‑media videos.

In summary, MS‑Mix offers a principled, emotion‑sensitive augmentation strategy that mitigates label noise, respects semantic consistency, and adaptively balances multimodal contributions, setting a new benchmark for robust multimodal sentiment analysis.