Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

While open sourced Vision-Language Models (VLMs) have proliferated, selecting the optimal pretrained model for a specific downstream task remains challenging. Exhaustive evaluation is often infeasible due to computational constraints and data limitations in few shot scenarios. Existing selection methods fail to fully address this: they either rely on data-intensive proxies or use symmetric textual descriptors that neglect the inherently directional and model-specific nature of transferability. To address this problem, we propose a framework that grounds model selection in the internal functional dynamics of the visual encoder. Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment. Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target’s salient functional blocks. This allows for predicting target model rankings by aggregating source task ranks without direct inference. Experimental results on 48 VLMs across 21 datasets demonstrate that our method outperforms state-of-the-art baselines, achieving a 14.7% improvement in NDCG@5 over SWAB.

💡 Research Summary

The paper tackles the practical problem of selecting the most suitable pretrained Vision‑Language Model (VLM) for a downstream task when only a handful of unlabeled images are available. Existing selection methods fall into two camps: data‑heavy proxies (e.g., H‑Score, LogME, LEEP) that require many labeled target samples, and text‑only approaches (LO‑VM, SWAB) that ignore the visual signal and treat task similarity as a symmetric, model‑agnostic notion. Both are inadequate for the few‑shot or zero‑shot regimes that dominate real‑world VLM deployment.

To overcome these limitations, the authors propose a model‑specific task representation based on layer conductance—an attribution technique that quantifies how much each internal block of a neural network contributes to a scalar objective. For a given VLM m, its visual encoder is partitioned into dₘ coarse‑grained functional blocks (e.g., ResNet stages or ViT transformer blocks). For an input image x, the conductance vector gₘ(x) =