Invariant Representation Guided Multimodal Sentiment Decoding with Sequential Variation Regularization

Achieving consistent sentiment representation across diverse modalities remains a key challenge in multimodal sentiment analysis. However, rapid emotional fluctuations over time often introduce instability, leading to compromised prediction performance. To address this challenge, we propose a robust sentiment representation dual enhancement strategy that simultaneously enhances the temporal and modality dimensions, guided by targeted mechanisms in both forward and backward propagation. Specifically, in the modality dimension, we introduce a modality invariant fusion mechanism that fosters stable cross-modal representations, which aim to capture the common and stable representations shared across different modalities. In the temporal dimension, we impose a specialized sequential variation regularization term that regulates the model’s learning trajectory during backward propagation, which is essentially total variation regularization degenerated into one-dimensional linear differences. Extensive experiments on three standard public datasets validate the effectiveness of our proposed approach.

💡 Research Summary

The paper tackles the persistent problem of unstable sentiment representations in multimodal sentiment analysis (MSA), especially when rapid emotional fluctuations occur within a video. To achieve robust sentiment decoding, the authors propose a dual‑enhancement framework that simultaneously strengthens the modality dimension and the temporal dimension.

In the modality dimension, each modality (text, audio, video) is first encoded (RoBERTa for text, a one‑layer Transformer for audio and video) to obtain raw features H_i. A shared encoder extracts modality‑invariant representations I_i, while private encoders produce modality‑specific representations S_i. Consistency between invariant representations is enforced with a Central Moment Discrepancy (CMD) loss (L_con). An adversarial discriminator, combined with a gradient‑reversal layer, pushes invariant and specific representations apart, and an additive angular margin loss (L_am) yields a domain loss (L_dom) that encourages clear modality separation.

For the temporal dimension, the authors introduce Sequential Variation Regularization (SVR). They approximate total variation (TV) regularization by a one‑dimensional form and measure the distance between adjacent frame distributions using Jensen–Shannon Divergence (JSD). The resulting temporal‑invariant loss L_ti = Σ_i JSD(softmax(R_i), softmax(R_{i+1})) penalizes abrupt changes in the video representation sequence, thereby smoothing rapid sentiment spikes.

The fusion module is guided by the invariant representations. Factorized Bilinear Pooling (FBP) generates gating signals (Sign_a, Sign_v) from invariant features, which modulate cross‑attention outputs between text and each of the other modalities. The final fused vector is the concatenation of gated audio and video features, which is fed to an MLP for prediction.

The overall training objective combines task loss (regression or classification), L_con, L_dom, and L_ti with trade‑off coefficients α=1.0, β=0.4, γ=1.0. Experiments on three benchmarks—CMU‑MOSI, CMU‑MOSEI, and UR‑FUNNY—show that the proposed method outperforms recent state‑of‑the‑art baselines (MISA, MMIN, TFN, etc.) across most metrics, achieving higher binary and seven‑class accuracies, F1 scores, and Pearson correlations. Ablation studies confirm that each component (adversarial learning, SVR, invariant‑guided gating) contributes positively; removing SVR notably degrades performance on samples with rapid emotional shifts.

Robustness tests add Gaussian noise with varying standard deviations (0.1, 0.5, 1.0) to the extracted features. The model’s performance remains stable, and in some cases improves with higher‑variance noise, suggesting that SVR provides a regularizing effect that emphasizes global temporal patterns over fine‑grained noisy details.

Visualization of loss curves demonstrates consistent reduction of L_con, L_dom, and L_ti during training, indicating effective optimization of both modality alignment and temporal smoothness.

Overall, the paper presents a coherent strategy that leverages invariant representations for cross‑modal alignment and a simplified total‑variation regularizer for temporal consistency. This combination yields a sentiment decoder that is both modality‑agnostic and resilient to rapid emotional fluctuations, making it suitable for real‑time affective computing applications.