OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts

The ability to segment objects based on open-ended language prompts remains a critical challenge, requiring models to ground textual semantics into precise spatial masks while handling diverse and unseen categories. We present OpenWorldSAM, a framework that extends the prompt-driven Segment Anything Model v2 (SAM2) to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). Our approach is guided by four key principles: i) Unified prompting: OpenWorldSAM supports a diverse range of prompts, including category-level and sentence-level language descriptions, providing a flexible interface for various segmentation tasks. ii) Efficiency: By freezing the pre-trained components of SAM2 and the VLM, we train only 4.5 million parameters on the COCO-stuff dataset, achieving remarkable resource efficiency. iii) Instance Awareness: We enhance the model’s spatial understanding through novel positional tie-breaker embeddings and cross-attention layers, enabling effective segmentation of multiple instances. iv) Generalization: OpenWorldSAM exhibits strong zero-shot capabilities, generalizing well on unseen categories and an open vocabulary of concepts without additional training. Extensive experiments demonstrate that OpenWorldSAM achieves state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across multiple benchmarks. Code is available at https://github.com/GinnyXiao/OpenWorldSAM.

💡 Research Summary

OpenWorldSAM proposes a lightweight yet powerful extension of the Segment Anything Model v2 (SAM2) that enables open‑vocabulary image segmentation driven solely by language prompts. The authors identify two fundamental shortcomings of existing promptable segmentation models: (1) lack of semantic grounding for free‑form text, and (2) inability to handle multiple instances that match the same textual query. To address these, they integrate a frozen vision‑language encoder (BEiT‑3) with the frozen SAM2 backbone and train only a 4.5 million‑parameter adapter consisting of a two‑layer MLP projector, learnable positional “tie‑breaker” vectors, and a three‑layer soft‑prompt Transformer.

The workflow begins by feeding both the image and the textual description into BEiT‑3, which jointly encodes them and produces a CLS token that captures the prompt’s semantics in the context of the image. This 1024‑dimensional vector is projected to 256 dimensions to match SAM2’s prompt channel size. To enable multi‑instance segmentation, K learnable positional vectors (typically K = 20) are added to the projected embedding, creating K distinct query vectors. Each query is then refined through a soft‑prompt Transformer that alternates self‑attention (promoting diversity among queries) and cross‑attention with SAM2’s level‑3 feature map (64 × 64 resolution). This cross‑attention grounds the language‑aware queries in high‑resolution visual features, effectively disambiguating overlapping objects.

The refined queries replace conventional point or box prompts in SAM2’s prompt encoder, and the mask decoder produces K masks together with confidence scores. During training, all heavy components (SAM2’s hierarchical vision transformer and BEiT‑3) remain frozen. Ground‑truth masks for the target class are matched to the K predictions using Hungarian matching, and a combination of focal loss and mask‑IoU loss is applied. This scheme forces each positional tie‑breaker to specialize in a different spatial region without any explicit supervision on the number of instances.

At inference time, the K masks are post‑processed for three downstream tasks: (i) semantic segmentation – masks of the same class are merged weighted by confidence, (ii) instance segmentation – low‑confidence masks are filtered out and non‑maximum suppression removes duplicates, and (iii) panoptic segmentation – a combination of the previous steps yields a unified “thing”/“stuff” labeling. An optional two‑stage refinement can feed the first‑stage masks back as visual prompts to SAM2 for finer boundary detail, though quantitative gains are modest.

Extensive experiments on six benchmarks—including COCO‑Stuff, ADE20K, Pascal‑Context, SUN‑RGBD, PC‑Seg, and RefCOCOg—demonstrate that OpenWorldSAM achieves state‑of‑the‑art zero‑shot performance. Notably, it reaches 60.4 mIoU on ADE20K and 74.0 cIoU on RefCOCOg, surpassing prior lightweight approaches such as EVF‑SAM and LISA while using far fewer trainable parameters. The model also shows strong generalization to unseen categories, confirming the effectiveness of the early‑fusion BEiT‑3 encoder and the positional tie‑breaker mechanism.

The paper discusses limitations: the fixed number of tie‑breaker vectors may be insufficient for scenes with many instances, and the reliance on a pre‑trained BEiT‑3 means that truly “zero‑resource” scenarios could suffer. Future work could explore dynamic instance count estimation, more adaptive tie‑breaker generation, or training a compact vision‑language encoder from scratch to further reduce dependency on large external models.

In summary, OpenWorldSAM successfully merges SAM2’s high‑quality mask generation with a compact language adapter, delivering a unified interface for semantic, instance, panoptic, and referring‑expression segmentation using only textual prompts. Its efficiency, strong zero‑shot capabilities, and minimal additional parameters make it a promising foundation for interactive applications, robotics, AR/VR, and other real‑time systems that require flexible, open‑world visual understanding.