SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation

Image segmentation plays an important role in vision understanding. Recently, the emerging vision foundation models continuously achieved superior performance on various tasks. Following such success, in this paper, we prove that the Segment Anything Model 2 (SAM2) can be a strong encoder for U-shaped segmentation models. We propose a simple but effective framework, termed SAM2-UNet, for versatile image segmentation. Specifically, SAM2-UNet adopts the Hiera backbone of SAM2 as the encoder, while the decoder uses the classic U-shaped design. Additionally, adapters are inserted into the encoder to allow parameter-efficient fine-tuning. Preliminary experiments on various downstream tasks, such as camouflaged object detection, salient object detection, marine animal segmentation, mirror detection, and polyp segmentation, demonstrate that our SAM2-UNet can simply beat existing specialized state-of-the-art methods without bells and whistles. Project page: \url{https://github.com/WZH0120/SAM2-UNet}.

💡 Research Summary

**
The paper introduces SAM2‑UNet, a straightforward yet powerful U‑shaped segmentation framework that leverages the hierarchical Hiera backbone of Segment Anything Model 2 (SAM2) as its encoder and couples it with a classic U‑Net decoder. Recognizing that SAM2, while impressive as a foundation model, only produces class‑agnostic masks when no prompts are supplied, the authors aim to adapt its rich feature extraction capabilities for a wide range of downstream segmentation tasks, both natural and medical.

Key architectural choices include: (1) using the pre‑trained Hiera backbone (Hiera‑L, Base+, etc.) which provides multi‑scale token representations through hierarchical pooling and window‑based attention, thereby overcoming the limitations of the flat ViT encoder used in SAM1; (2) freezing the massive Hiera parameters (≈214 M) and inserting lightweight adapters before each hierarchical block. Each adapter consists of a down‑projection linear layer, a GeLU activation, an up‑projection linear layer, and a second GeLU, allowing parameter‑efficient fine‑tuning with only a tiny fraction of the total weights. This design makes the model feasible on a single RTX 4090 with 24 GB memory.

The decoder follows the traditional U‑Net design: three decoder blocks, each containing two Conv‑BN‑ReLU sequences and a 1×1 convolution that produces intermediate segmentation maps (S₁, S₂, S₃). Deep supervision is applied to all three outputs, and the loss combines weighted IoU with binary cross‑entropy, encouraging both region‑level overlap and pixel‑wise accuracy.

Experiments span five major benchmarks—camouflaged object detection, salient object detection, marine animal segmentation, mirror detection, and polyp segmentation—covering a total of eighteen public datasets. The authors compare against a broad set of state‑of‑the‑art methods (e.g., SINet, PFNet, ZoomNet, FEDER, U2Net, ICON, EDN, MENet, C2FNet, OCENet, MASNet, MirrorNet, HetNet, PraNet, SANet, CaraNet, CF A‑Net). Across all metrics, SAM2‑UNet consistently outperforms these specialized models. Notable results include S‑measure 0.914 on the CHAMELEON camouflaged‑object dataset, mDice 0.928 on Kvasir‑SEG polyp data, and IoU 0.918 on the MSD mirror detection set. Visual examples demonstrate superior boundary recovery and reduced false positives/negatives in challenging scenes.

Ablation studies examine the impact of backbone size. Progressively larger Hiera variants (Tiny → Small → Base+ → Large) yield steady performance gains, confirming that the richer representations from larger pre‑trained backbones translate into better downstream results, even when only adapters are trained.

In conclusion, SAM2‑UNet delivers on three promises: simplicity (classic U‑Net architecture, easy to implement), efficiency (adapter‑based fine‑tuning, low memory footprint), and effectiveness (state‑of‑the‑art performance across diverse domains). The work establishes a new baseline for future SAM2‑based segmentation research and suggests promising extensions such as multimodal prompting, lightweight mobile deployment, and further architectural refinements.