Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing

Recent advances in image understanding have enabled methods that leverage large language models for multimodal reasoning in remote sensing. However, existing approaches still struggle to steer models to the user-relevant regions when only simple, generic text prompts are available. Moreover, in large-scale aerial imagery many objects exhibit highly similar visual appearances and carry rich inter-object relationships, which further complicates accurate recognition. To address these challenges, we propose Cross-modal Context-aware Learning for Visual Prompt-Guided Multimodal Image Understanding (CLV-Net). CLV-Net lets users supply a simple visual cue, a bounding box, to indicate a region of interest, and uses that cue to guide the model to generate correlated segmentation masks and captions that faithfully reflect user intent. Central to our design is a Context-Aware Mask Decoder that models and integrates inter-object relationships to strengthen target representations and improve mask quality. In addition, we introduce a Semantic and Relationship Alignment module: a Cross-modal Semantic Consistency Loss enhances fine-grained discrimination among visually similar targets, while a Relationship Consistency Loss enforces alignment between textual relations and visual interactions. Comprehensive experiments on two benchmark datasets show that CLV-Net outperforms existing methods and establishes new state-of-the-art results. The model effectively captures user intent and produces precise, intention-aligned multimodal outputs.

💡 Research Summary

This paper addresses the significant challenges in multimodal understanding of remote sensing (RS) imagery, where objects often exhibit high visual similarity and exist within complex inter-object relationships, making accurate interpretation difficult. Existing methods that rely solely on generic textual prompts struggle to focus on user-specific regions of interest and to disambiguate between visually akin objects. To overcome these limitations, the authors propose CLV-Net (Cross-modal Context-aware Learning for Visual Prompt-Guided Multimodal Image Understanding), a novel framework that leverages intuitive visual prompts for precise, user-intent-aligned output generation.

The core innovation of CLV-Net is its interaction paradigm. Instead of requiring users to formulate detailed textual descriptions, CLV-Net allows them to provide a simple visual prompt—a bounding box—directly on the image to specify a region of interest. The model then uses this cue to generate a hierarchical textual caption (a global summary plus detailed local descriptions centered on the prompted region) alongside corresponding segmentation masks for each object phrase mentioned in the caption. This significantly reduces user effort while ensuring the output is tightly focused on user intent.

The CLV-Net architecture consists of three principal components designed to work cohesively:

1. Visual-Prompt Scene Reasoner (VPReasoner): This module takes the input image, the visual bounding box prompt, and a global textual prompt. It fuses these multimodal cues and conditions a Large Language Model (LLM) to generate the hierarchical captions that contextualize the user-specified area within the broader scene.
2. Context-Aware Mask Decoder (CMDecoder): This decoder is responsible for producing high-quality segmentation masks that align with the object phrases in the generated caption. Its key element is a Context-aware Graph Former, which explicitly models semantic relationships between objects. It constructs a graph where nodes represent object embeddings from the text, and uses multi-head cross-attention to compute a relationship matrix. This contextual information is then integrated to refine the visual features of each object, leading to more accurate and contextually aware mask predictions, especially for distinguishing between visually similar entities.
3. Semantic and Relationship Alignment Module (SRAlign): To bridge the gap between textual and visual representations and ensure precise cross-modal correspondence, CLV-Net introduces two novel loss functions during training. The Cross-modal Semantic Consistency Loss employs a contrastive learning strategy to pull together the feature representations of the same object across text and vision modalities while pushing apart representations of different objects, enhancing fine-grained discriminative power. The Relationship Consistency Loss enforces alignment between the relational structures in the textual domain (e.g., “next to”) and the similarity relationships among visual object features, ensuring that semantic relationships are consistently reflected in the visual output.

Comprehensive evaluations on the remote sensing benchmark dataset GeoPixelD and the natural image dataset GranD demonstrate that CLV-Net establishes new state-of-the-art performance. It outperforms existing methods like GeoPixel, GeoPix, and GLaMM in metrics evaluating caption quality, mask accuracy, and the alignment between generated text and masks. The results validate the effectiveness of the visual prompt guidance, the context-aware relational modeling in the mask decoder, and the proposed cross-modal alignment losses.

In summary, CLV-Net presents a significant advancement in multimodal RS image understanding by introducing an intuitive visual prompting mechanism and a sophisticated architecture that deeply models object context and enforces cross-modal consistency. It offers a more user-friendly and accurate approach for generating intention-aligned, comprehensive descriptions and segmentations of complex aerial and satellite imagery.