Generalization vs. Specialization: Evaluating Segment Anything Model (SAM3) Zero-Shot Segmentation Against Fine-Tuned YOLO Detectors

Deep learning has advanced two fundamentally different paradigms for instance segmentation: specialized models optimized through task-specific fine-tuning and generalist foundation models capable of zero-shot segmentation. This work presents a comprehensive comparison between SAM3 (Segment Anything Model, also called SAMv3) operating in zero-shot mode and three variants of Ultralytics YOLO11 (nano, medium, and large) fine-tuned for instance segmentation. The evaluation is conducted on the MinneApple dataset, a dense benchmark comprising 670 orchard images with 28,179 annotated apple instances, enabling rigorous validation of model behavior under high object density and occlusion. Our analysis shows IoU choices can inflate performance gaps by up to 30%. At the appropriate IoU = 0.15 threshold, YOLO models achieve 68.9%, 72.2%, and 71.9% F1, while SAM3 reaches 59.8% in pure zero-shot mode. However, YOLO exhibits steep degradation 48-50 points across IoU ranges whereas SAM3 drops only 4 points, revealing 12 times superior boundary stability of SAM3. This highlights the strength of SAMv3 in mask precision versus specialization in detection completeness of YOLO11. We provide open-source code, evaluation pipelines, and methodological recommendations, contributing to a deeper understanding of when specialized fine-tuned models or generalist foundation models are preferable for dense instance segmentation tasks. This project repository is available on GitHub as https://github.com/Applied-AI-Research-Lab/Segment-Anything-Model-SAM3-Zero-Shot-Segmentation-Against-Fine-Tuned-YOLO-Detectors

💡 Research Summary

This paper presents a rigorous comparative study between two divergent paradigms in instance segmentation: specialized models optimized through task-specific fine-tuning and generalist foundation models capable of zero-shot segmentation. The research specifically evaluates the performance of SAM3 (Segment Anything Model v3) in a zero-shot configuration against three variants of the Ultralytics YOLO11 architecture (nano, medium, and large) fine-tuned for instance segmentation.

The evaluation was conducted using the MinneApple dataset, a challenging benchmark characterized by high object density and significant occlusion, containing 28,179 annotated apple instances across 670 orchard images. This dataset provides a robust environment for testing model reliability in complex, real-world agricultural scenarios.

The core findings of the study reveal a critical trade-off between “detection completeness” and “mask precision.” At a low IoU (Intersection over Union) threshold of 0.15, the fine-tuned YOLO11 models demonstrated superior performance, achieving F1 scores ranging from 68.9% to 72.2%, significantly outperforming SAM3’s 59.8%. This indicates that specialized models remain highly effective at identifying and localizing objects within a specific domain.

However, the study uncovers a profound disparity in performance stability as the IoU threshold increases. The YOLO11 models exhibited a dramatic performance collapse, with F1 scores dropping by 48 to 50 points across higher IoU ranges. In stark contrast, SAM3 demonstrated remarkable robustness, with its performance declining by only 4 points. This indicates that SAM3 possesses 12 times superior boundary stability compared to the specialized YOLO models.

The researchers conclude that the choice between these two paradigms should be driven by the specific requirements of the task. YOLO11 is the preferred choice when the primary objective is high recall and detection completeness in dense environments. Conversely, SAM3 is superior when the priority is high-fidelity mask precision and structural consistency of object boundaries. By quantifying the sensitivity of IoU thresholds and the stability of segmentation masks, this work provides essential methodological recommendations for deploying segmentation models in high-density instance segmentation tasks.