Bridging Lexical Ambiguity and Vision: A Mini Review on Visual Word Sense Disambiguation

This paper offers a mini review of Visual Word Sense Disambiguation (VWSD), which is a multimodal extension of traditional Word Sense Disambiguation (WSD). VWSD helps tackle lexical ambiguity in vision-language tasks. While conventional WSD depends only on text and lexical resources, VWSD uses visual cues to find the right meaning of ambiguous words with minimal text input. The review looks at developments from early multimodal fusion methods to new frameworks that use contrastive models like CLIP, diffusion-based text-to-image generation, and large language model (LLM) support. Studies from 2016 to 2025 are examined to show the growth of VWSD through feature-based, graph-based, and contrastive embedding techniques. It focuses on prompt engineering, fine-tuning, and adapting to multiple languages. Quantitative results show that CLIP-based fine-tuned models and LLM-enhanced VWSD systems consistently perform better than zero-shot baselines, achieving gains of up to 6-8% in Mean Reciprocal Rank (MRR). However, challenges still exist, such as limitations in context, model bias toward common meanings, a lack of multilingual datasets, and the need for better evaluation frameworks. The analysis highlights the growing overlap of CLIP alignment, diffusion generation, and LLM reasoning as the future path for strong, context-aware, and multilingual disambiguation systems.

💡 Research Summary

This mini‑review surveys the emerging field of Visual Word Sense Disambiguation (VWSD), a multimodal extension of traditional Word Sense Disambiguation (WSD) that leverages visual cues to resolve lexical ambiguity when only minimal textual context is available. The authors systematically collected peer‑reviewed papers from 2016 to 2025 across major venues (ACL Anthology, arXiv, IEEE Xplore, CVPR, etc.) using a PRISMA‑style flow, ultimately focusing on works that report empirical results on the SemEval‑2023 Task 1 benchmark or comparable multimodal datasets.

The review first outlines the historical progression from early feature‑based and graph‑based fusion methods (2016‑2019). Feature‑based approaches combined CNN visual embeddings (VGG, ResNet) with static word embeddings (Word2Vec, GloVe) via simple concatenation, attention mechanisms, or Canonical Correlation Analysis (CCA). Graph‑based techniques modeled candidate images and senses as nodes, constructing multimodal similarity graphs and applying label‑propagation, random‑walk, or Graph Convolutional Networks (GCNs) to spread sense information, achieving notable gains especially for verb‑sense tasks (VVSD).

A major turning point arrived with Contrastive Language‑Image Pre‑training (CLIP). Zero‑shot CLIP computes cosine similarity between text and image embeddings, allowing direct disambiguation without task‑specific training. The authors emphasize the critical role of prompt engineering: carefully crafted prompts (e.g., “A photo of a bat used in sports” vs. “A bat hanging upside down”) steer CLIP toward the intended sense and can improve Mean Reciprocal Rank (MRR) by several points. Fine‑tuning strategies are categorized into full‑parameter adaptation and lightweight adapters inserted into CLIP’s visual and textual encoders; both approaches consistently outperform the zero‑shot baseline, with fine‑tuned CLIP achieving 5‑7 % absolute MRR gains, and CLIP‑BLIP hybrids reaching even higher scores.

Parallel to CLIP advances, large language models (LLMs) such as GPT‑3, OPT, and InstructGPT have been integrated to enrich the sparse textual context. Three main LLM‑driven techniques are highlighted: (1) Context‑Aware Definition Generation (CADG), where LLMs produce detailed sense definitions for ambiguous words, improving retrieval especially for out‑of‑vocabulary terms; (2) Context Expansion, where short noun phrases are transformed into richer descriptions containing synonyms, hypernyms, and related concepts, which are then fed to the multimodal encoder; and (3) Chain‑of‑Thought (CoT) prompting, enabling step‑by‑step reasoning that yields more interpretable disambiguation decisions. Empirical results show that LLM‑augmented systems add an extra 1‑2 % MRR over fine‑tuned CLIP, with pronounced benefits for low‑frequency senses and multilingual settings (Italian, Farsi).

The review also surveys emerging cross‑modal generation approaches that employ text‑to‑image diffusion models (DALL·E 2, Stable Diffusion) to generate candidate images from the ambiguous word’s definition and then compare these synthetic images with the provided options. Although promising, these methods remain in early experimental stages due to computational cost and evaluation challenges.

Despite steady progress, the authors identify persistent challenges: (i) limited textual context often leads models to default to the most common sense, causing bias toward prototypical meanings; (ii) multilingual resources are scarce, hindering robust LLM‑based expansion for non‑English languages; (iii) current evaluation relies heavily on ranking metrics (MRR, HIT@1), which may not capture downstream utility in tasks such as image retrieval or caption generation; and (iv) integration of diffusion‑based generation with contrastive retrieval lacks systematic benchmarking.

Future research directions proposed include (a) hybrid architectures that combine graph‑based label propagation with contrastive fine‑tuning to exploit both global structure and fine‑grained alignment; (b) unified multimodal prompting that jointly leverages CLIP, LLMs, and diffusion models for richer context and reasoning; (c) construction of large, multilingual multimodal sense‑annotated corpora and the development of more holistic evaluation frameworks that assess both ranking performance and downstream task impact.

In summary, the paper provides a comprehensive taxonomy of VWSD methods, documents quantitative gains achieved by CLIP fine‑tuning and LLM augmentation (up to 6‑8 % MRR improvement over zero‑shot baselines), and outlines the technical and resource challenges that must be addressed to realize robust, context‑aware, and multilingual visual sense disambiguation systems.