Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study

Manual labeling of animal images remains a significant bottleneck in ecological research, limiting the scale and efficiency of biodiversity monitoring efforts. This study investigates whether state-of-the-art Vision Transformer (ViT) foundation models can reduce thousands of unlabeled animal images directly to species-level clusters. We present a comprehensive benchmarking framework evaluating five ViT models combined with five dimensionality reduction techniques and four clustering algorithms, two supervised and two unsupervised, across 60 species (30 mammals and 30 birds), with each test using a random subset of 200 validated images per species. We investigate when clustering succeeds at species-level, where it fails, and whether clustering within the species-level reveals ecologically meaningful patterns such as sex, age, or phenotypic variation. Our results demonstrate near-perfect species-level clustering (V-measure: 0.958) using DINOv3 embeddings with t-SNE and supervised hierarchical clustering methods. Unsupervised approaches achieve competitive performance (0.943) while requiring no prior species knowledge, rejecting only 1.14% of images as outliers requiring expert review. We further demonstrate robustness to realistic long-tailed distributions of species and show that intentional over-clustering can reliably extract intra-specific variation including age classes, sexual dimorphism, and pelage differences. We introduce an open-source benchmarking toolkit and provide recommendations for ecologists to select appropriate methods for sorting their specific taxonomic groups and data.

💡 Research Summary

**
The paper presents a comprehensive benchmark of Vision Transformer (ViT) foundation models for zero‑shot clustering of animal images, addressing the critical bottleneck of manual labeling in biodiversity monitoring. Using a curated dataset of 60 species (30 mammals, 30 birds) drawn from 23 camera‑trap projects, the authors assembled 139,111 validated animal crops (≈200 images per species for testing) that reflect realistic long‑tailed species distributions. Five state‑of‑the‑art ViT models—DINOv2, DINOv3, CLIP‑ViT‑B/16, BioCLIP, and BioCLIP‑2—were evaluated. Each model’s high‑dimensional embeddings (768–1024 dimensions) were projected into low‑dimensional space with five dimensionality‑reduction techniques (PCA, UMAP, t‑SNE, PHATE, Isomap). The reduced embeddings were then clustered using four algorithms: two supervised (hierarchical clustering, K‑means) and two unsupervised (HDBSCAN, Gaussian Mixture Models).

Key findings include:

• The combination of DINOv3 embeddings, t‑SNE projection, and hierarchical clustering achieved near‑perfect species‑level separation (V‑measure = 0.958, accuracy ≈ 96 %).
• Unsupervised HDBSCAN performed competitively (V‑measure = 0.943) while flagging only 1.14 % of images as outliers, dramatically reducing expert review workload.
• Both supervised and unsupervised pipelines remained robust under long‑tailed conditions, correctly clustering rare species with as few as 400 images.
• Intentional over‑clustering (setting the number of clusters higher than the true species count) revealed biologically meaningful sub‑clusters corresponding to age classes, sexual dimorphism, and seasonal pelage changes, demonstrating the method’s capacity to uncover intra‑specific variation without additional annotation.

The study also identifies practical constraints: t‑SNE’s computational cost limits scalability to very large datasets; some species suffer from background‑induced embedding mixing, indicating a need for better foreground focus or domain‑adaptation; and density‑based unsupervised methods are sensitive to hyper‑parameters, necessitating automated tuning.

To facilitate adoption, the authors release an open‑source benchmarking toolkit and an interactive web visualization platform, allowing ecologists to explore model‑reduction‑clustering combinations without programming expertise. Based on empirical results, they provide concrete deployment guidelines: for large, diverse datasets prioritize DINOv3‑t‑SNE‑hierarchical clustering; for rapid, label‑free workflows favor HDBSCAN; and for studies interested in fine‑grained phenotypic variation, employ intentional over‑clustering.

In conclusion, the paper demonstrates that modern ViT embeddings can effectively organize unlabeled animal imagery into species‑level clusters, substantially lowering annotation costs while preserving the ability to detect ecologically relevant intra‑species patterns. This work bridges the gap between proof‑of‑concept zero‑shot clustering and real‑world ecological applications, offering a scalable, reproducible pipeline for biodiversity monitoring programs worldwide.