Leveraging Textual-Cues for Enhancing Multimodal Sentiment Analysis by Object Recognition

Multimodal sentiment analysis, which includes both image and text data, presents several challenges due to the dissimilarities in the modalities of text and image, the ambiguity of sentiment, and the complexities of contextual meaning. In this work, we experiment with finding the sentiments of image and text data, individually and in combination, on two datasets. Part of the approach introduces the novel `Textual-Cues for Enhancing Multimodal Sentiment Analysis’ (TEMSA) based on object recognition methods to address the difficulties in multimodal sentiment analysis. Specifically, we extract the names of all objects detected in an image and combine them with associated text; we call this combination of text and image data TEMS. Our results demonstrate that only TEMS improves the results when considering all the object names for the overall sentiment of multimodal data compared to individual analysis. This research contributes to advancing multimodal sentiment analysis and offers insights into the efficacy of TEMSA in combining image and text data for multimodal sentiment analysis.

💡 Research Summary

The paper addresses the challenge of jointly analyzing sentiment from images and accompanying text, a task complicated by the heterogeneous nature of visual and linguistic modalities and the inherent ambiguity of sentiment. The authors propose a novel framework called TEMSA (Textual‑Cues for Enhancing Multimodal Sentiment Analysis) that leverages state‑of‑the‑art object detection (DETR and Faster‑R‑CNN) to extract the names of all detectable objects in an image. These object names are then concatenated with the image’s caption or super‑imposed text, forming a single textual sequence referred to as TEMS (Text‑Enhanced Multimodal Sentiment). By converting visual cues into textual tokens, standard language‑based sentiment classifiers—BiLSTM and BERT—can directly ingest both modalities without the need for separate visual feature pipelines.

The authors evaluate the approach on two publicly available social‑media datasets. The SIMPSoN dataset consists of Instagram news‑related images with manually annotated sentiment labels for image‑only, text‑only, and joint (image + text) perspectives. After filtering out non‑English captions, 2,830 samples remain. The MVSA‑Single dataset contains Twitter image‑text pairs; the authors create a new joint sentiment label by retaining only those pairs where image and text share the same sentiment, yielding 486 samples.

Four experimental conditions are explored: (1) visual‑only sentiment prediction using deep convolutional networks (VGG‑16, ResNet) and Vision Transformer (ViT); (2) text‑only prediction using BiLSTM and BERT; (3) multimodal prediction where the TEMS string is fed to the same text models; and (4) a subset experiment where only a single object is detected per image, to assess the contribution of multiple object names. Results consistently show that the TEMS‑augmented models outperform both unimodal baselines, achieving an average improvement of 3–5 percentage points in accuracy. The gain is most pronounced for images with two or more detected objects, confirming that aggregating all object names provides richer contextual cues than a single‑object representation.

The paper also discusses related work, noting that prior multimodal sentiment studies either process whole‑image visual features, focus on a limited set of regions, or simply concatenate coarse visual embeddings with text. By contrast, TEMSA treats every detected object as a textual token, thereby preserving fine‑grained visual semantics in a format naturally handled by language models. Limitations include potential noise from irrelevant object names, dependence on the quality of the object detector, and the lack of explicit semantic alignment between object tokens and sentiment‑bearing words. Future directions suggested are semantic filtering of object names (e.g., synonym clustering), attention‑based fusion of visual and textual streams, and incorporation of non‑standard textual cues such as emojis or hashtags.

In summary, the study demonstrates that converting visual object detections into textual cues and merging them with existing captions yields a simple yet effective multimodal sentiment analysis pipeline, outperforming traditional separate‑modality approaches and opening new avenues for integrating visual semantics into language‑centric models.