Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialogue research.

💡 Research Summary

The paper introduces Image‑Grounded Conversations (IGC), a novel task that requires generating natural, multi‑turn dialogues anchored on a shared image. Unlike traditional image captioning or Visual Question Answering (VQA), where the visual input is directly described or queried, IGC treats the image as a contextual catalyst that shapes the topic and flow of conversation. The task is split into two sub‑tasks: (1) Question Generation – given an image I and an initial textual context T (e.g., a user’s opening remark about the picture), produce a coherent, engaging question Q that naturally follows; (2) Response Generation – given I, T, and the generated (or human‑provided) question Q, produce an appropriate response R. The questions are deliberately not answerable solely from the image, emphasizing inference and engagement rather than factual retrieval.

To benchmark progress, the authors construct a high‑quality, crowd‑sourced dataset called IGC‑Crowd. They first select event‑centric images from the VQG dataset using event‑related search queries. Using Amazon Mechanical Turk, pairs of workers engage in real‑time chat about a chosen image, producing three‑turn dialogues (initial statement, question, response). For each dialogue, five additional question‑response pairs are collected, yielding multi‑reference data. The final corpus contains 4,222 conversations (25,332 utterances) and 42,220 reference utterances, split 40 %/60 % for validation and testing. Additionally, a large auxiliary training set of 250 K three‑turn Twitter threads (1.4 M tweets) is harvested to pre‑train models.

The authors analyze the dataset from several angles. Human judges rate the “effectiveness” of visual versus textual context for forming questions; both modalities are found highly necessary, with textual context playing a slightly larger role in the crowd‑sourced set. FrameNet annotations reveal that many questions rely more on the textual frame than the visual frame, confirming the complementary nature of the two modalities. Causal and temporal event annotations (using the CaTERS scheme) show that each utterance mentions on average 0.71 event entities and that rich commonsense relations are pervasive, underscoring the need for models that can capture event dynamics, not just object recognition.

Three neural generation architectures are evaluated. All use VGG‑19 to extract a 4096‑dimensional image vector (fc7). The first model concatenates image and text embeddings and feeds them to a standard Seq2Seq LSTM. The second injects the image vector as the initial hidden state of the decoder LSTM (image‑conditioned LSTM). The third adds a multimodal attention mechanism that dynamically weights image and text features at each decoding step. These models are first trained on the large Twitter corpus, then fine‑tuned on the IGC‑Crowd data.

Automatic metrics (BLEU, METEOR, ROUGE) and human evaluations are reported. The multimodal attention model consistently outperforms the other two, confirming that explicit attention to both visual and textual cues improves fluency and relevance. Nevertheless, a substantial gap remains between model outputs and human‑generated utterances, especially in terms of engaging, context‑aware questioning and nuanced response generation. Human judges note that current systems struggle with multimodal anaphora resolution, event causality, and maintaining coherent conversational flow.

In conclusion, the paper defines a new, challenging dialogue generation task that sits between open‑ended chit‑chat and goal‑oriented dialog, introduces a publicly released, richly annotated dataset, and provides baseline neural models that demonstrate the benefit of multimodal context. The work opens several avenues for future research: more sophisticated multimodal encoders (e.g., Transformers with cross‑modal attention), incorporation of explicit event graphs or commonsense knowledge bases, and reinforcement‑learning approaches to optimize conversational objectives such as user engagement or information elicitation.