VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images

Interactive image segmentation(IIS) plays a critical role in generating precise annotations for remote sensing imagery, where objects often exhibit scale variations, irregular boundaries and complex backgrounds. However, existing IIS methods, primarily designed for natural images, struggle to generalize to remote sensing domains due to limited annotated data and computational overhead. To address these challenges, we proposed RS-ISRefiner, a novel click-based IIS framework tailored for remote sensing images. The framework employs an adapter-based tuning strategy that preserves the general representations of Vision Foundation Models while enabling efficient learning of remote sensing-specific spatial and boundary characteristics. A hybrid attention mechanism integrating convolutional local modeling with Transformer-based global reasoning enhances robustness against scale diversity and scene complexity. Furthermore, an improved probability map modulation scheme effectively incorporates historical user interactions, yielding more stable iterative refinement and higher boundary accuracy. Comprehensive experiments on six remote sensing datasets, including iSAID, ISPRS Potsdam, SandBar, NWPU, LoveDA Urban and WHUBuilding, demonstrate that RS-ISRefiner consistently outperforms state-of-the-art IIS methods in terms of segmentation accuracy, efficiency and interaction cost. These results confirm the effectiveness and generalizability of our framework, making it highly suitable for high-quality instance segmentation in practical remote sensing scenarios. The codes are available at https://github.com/wondelyan/VFM-ISRefiner .

💡 Research Summary

The paper “VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images” addresses the significant challenges of applying interactive image segmentation (IIS) to remote sensing (RS) imagery. RS images present unique difficulties such as extreme scale variations among objects (e.g., small vehicles vs. large fields), irregular and complex boundaries (e.g., river sandbars, shadowed buildings), and cluttered spectral-spatial backgrounds. Existing IIS methods, predominantly designed for natural images, struggle with these domain-specific characteristics, often requiring excessive user corrective clicks (high interaction cost) and suffering from performance degradation due to the domain gap.

To bridge this gap, the authors propose VFM-ISRefiner, a novel click-based IIS framework specifically tailored for RS images. The core innovation lies in efficiently adapting large-scale Vision Foundation Models (VFMs) like Vision Transformers (ViTs), which are pre-trained on massive natural image datasets, to the RS domain without costly full-parameter fine-tuning. The framework employs a three-pronged technical approach.

First, it uses an adapter-based tuning strategy. Instead of retraining the entire VFM backbone—which is computationally expensive and may overwrite valuable general-purpose visual knowledge—the backbone’s weights are kept frozen. Lightweight, trainable adapter modules are inserted into the network. These adapters are designed to efficiently learn RS-specific spatial and boundary characteristics, allowing the model to preserve its general representations while acquiring domain-adaptive capabilities.

Second, to handle the scale diversity and scene complexity of RS data, the framework incorporates a hybrid attention mechanism within its feature extractor. This mechanism synergistically combines convolutional operations, which excel at local spatial modeling and detailed edge feature extraction, with Transformer-based attention, which is powerful for global contextual reasoning. This hybrid design enhances the model’s robustness against the varied scales and cluttered backgrounds typical in RS scenes.

Third, the paper introduces an improved probability map modulation scheme for propagating historical user interaction information. During iterative refinement, previous segmentation probability maps are not just fed as input but are intelligently modulated. Building upon prior work (MFP), the proposed enhancement makes the modulation process more adaptable to the irregular shapes of RS objects, leading to more stable predictions across interaction rounds and higher final boundary accuracy, especially for geometrically complex targets.

The proposed VFM-ISRefiner was evaluated extensively on six diverse RS datasets: iSAID, ISPRS Potsdam, SandBar, NWPU, LoveDA Urban, and WHUBuilding. Comprehensive experiments demonstrated that it consistently outperforms state-of-the-art IIS methods (e.g., SimpleClick, FocalClick, MST, MFP) across key metrics, including segmentation accuracy (measured by mIoU), efficiency, and interaction cost (measured by the Number of Clicks, NoC, required to achieve a certain accuracy threshold). The results validate the effectiveness of the adapter-based adaptation, the hybrid attention design, and the enhanced history propagation mechanism. In conclusion, VFM-ISRefiner provides a powerful, efficient, and practical framework for achieving high-quality instance segmentation in RS applications with minimal user effort, effectively adapting the power of foundation models to a specialized domain. The code is publicly available for reproducibility and further research.