Towards Real Zero-Shot Camouflaged Object Segmentation without Camouflaged Annotations

Camouflaged Object Segmentation (COS) faces significant challenges due to the scarcity of annotated data, where meticulous pixel-level annotation is both labor-intensive and costly, primarily due to the intricate object-background boundaries. Addressing the core question, “Can COS be effectively achieved in a zero-shot manner without manual annotations for any camouflaged object?” we affirmatively respond and introduce a robust zero-shot COS framework. This framework leverages the inherent local pattern bias of COS and employs a broad semantic feature space derived from salient object segmentation (SOS) for efficient zero-shot transfer. We incorporate an Masked Image Modeling (MIM) based image encoder optimized for Parameter-Efficient Fine-Tuning (PEFT), a Multimodal Large Language Model (M-LLM), and a Multi-scale Fine-grained Alignment (MFA) mechanism. The MIM pre-trained image encoder focuses on capturing essential low-level features, while the M-LLM generates caption embeddings processed alongside these visual cues. These embeddings are precisely aligned using MFA, enabling our framework to accurately interpret and navigate complex semantic contexts. To optimize operational efficiency, we introduce a learnable codebook that represents the M-LLM during inference, significantly reducing computational overhead. Our framework demonstrates its versatility and efficacy through rigorous experimentation, achieving state-of-the-art performance in zero-shot COS with $F_β^w$ scores of 72.9% on CAMO and 71.7% on COD10K. By removing the M-LLM during inference, we achieve an inference speed comparable to that of traditional end-to-end models, reaching 18.1 FPS. Code: https://github.com/AVC2-UESTC/ZSCOS-CaMF

💡 Research Summary

The paper tackles the long‑standing challenge of Camouflaged Object Segmentation (COS), where objects blend seamlessly into their surroundings, making pixel‑level annotation both labor‑intensive and costly. The authors ask a bold question: can COS be performed in a truly zero‑shot manner, i.e., without any camouflaged annotations at all? Their answer is affirmative, and they introduce a comprehensive framework that combines several recent advances in vision‑language modeling, parameter‑efficient fine‑tuning, and multi‑scale feature alignment.

Key Components

1. Masked Image Modeling (MIM) Encoder – A vision backbone pre‑trained with a masked image modeling objective. This pre‑training forces the encoder to reconstruct missing patches, thereby learning rich low‑level textures and high‑frequency details that are crucial for delineating the subtle boundaries of camouflaged objects.
2. Parameter‑Efficient Fine‑Tuning (PEFT) – Instead of fine‑tuning the entire MIM encoder on a new dataset, the authors adopt PEFT (e.g., adapters or LoRA) to adjust only a small set of parameters while keeping the bulk of the pretrained weights frozen. They fine‑tune the model on a large Salient Object Segmentation (SOS) dataset, which provides abundant semantic context without requiring any camouflaged masks. This step injects global semantic awareness into the encoder without destroying its local sensitivity.
3. Multimodal Large Language Model (M‑LLM) – During training, an external multimodal LLM generates descriptive captions conditioned on the visual input (e.g., “a camouflaged fish hidden among coral”). The resulting text embeddings serve as high‑level semantic cues.
4. Multi‑scale Fine‑grained Alignment (MFA) – Visual features from multiple resolutions (low‑resolution global, high‑resolution local) are aligned with the caption embeddings via a cross‑attention mechanism. MFA ensures that each spatial scale receives appropriate semantic guidance, allowing the model to simultaneously capture fine boundary details and broader object context.
5. Learnable Codebook for Inference – Running the M‑LLM at inference is computationally prohibitive. To address this, the authors train a compact codebook that distills the essential semantic priors learned from the M‑LLM. During inference, the codebook’s discrete tokens replace the LLM, providing semantic prompts at negligible cost.

Training & Inference Workflow

• Training: The MIM encoder processes an image, producing a hierarchy of visual tokens. The M‑LLM generates a caption, which is encoded into a text token sequence. MFA aligns the two modalities, and the PEFT adapters are updated to minimize a segmentation loss computed against the SOS ground‑truth masks. Simultaneously, the codebook learns to approximate the LLM’s text embeddings.
• Inference: The image passes through the frozen MIM encoder and PEFT adapters. Instead of invoking the LLM, the model queries the learned codebook to retrieve a set of semantic tokens. These tokens are fused with the visual features via MFA, and a lightweight decoder predicts the camouflaged mask.

Results

• On the standard COS benchmarks CAMO and COD10K, the zero‑shot version achieves Fβʷ scores of 72.9 % and 71.7 % respectively, surpassing prior weakly‑supervised and open‑vocabulary methods that still rely on some camouflaged annotations.
• Inference speed reaches 18.1 FPS on an RTX 4060 Ti, dramatically faster than GenSAM (≈3.68 s per image) and MMCPF (≈3.45 s per image), thanks to the codebook substitution.
• The same architecture, without any modification, also excels in unrelated segmentation tasks such as polyp segmentation and underwater scene segmentation, demonstrating strong transferability.

Contributions

1. Proof of Concept – The work convincingly shows that COS can be tackled without any camouflaged training data, establishing a new baseline for truly annotation‑free dense prediction.
2. Methodological Innovation – By marrying a MIM‑trained local‑feature encoder with PEFT‑enhanced global semantics, and by aligning these with LLM‑derived textual cues via MFA, the authors create a balanced representation that satisfies both high‑frequency detail and semantic richness.
3. Efficiency via Codebook – The learnable codebook elegantly sidesteps the heavy computational load of large language models at test time, making the solution practical for real‑world deployment.

Limitations & Future Directions

• The codebook, while efficient, may not capture the full expressive power of the original LLM, potentially limiting performance on extremely nuanced semantic distinctions.
• Reliance on SOS data for global semantics could hinder generalization to domains where salient cues differ dramatically from natural images (e.g., medical imaging).
• Future work could explore dynamic codebooks that adapt online, incorporate multi‑domain SOS corpora, or integrate lightweight on‑device language models to further close the gap between efficiency and semantic fidelity.

Conclusion
The paper delivers a well‑engineered, experimentally validated framework that achieves state‑of‑the‑art zero‑shot camouflaged object segmentation without any camouflaged annotations. By thoughtfully combining MIM pre‑training, PEFT, multimodal language guidance, and a novel inference‑time codebook, it sets a new direction for annotation‑free dense prediction tasks across diverse visual domains.