VL-RouterBench Evaluating Vision-Language Model Routing Systems
📝 Original Paper Info
- Title: VL-RouterBench A Benchmark for Vision-Language Model Routing- ArXiv ID: 2512.23562
- Date: 2025-12-29
- Authors: Zhehao Huang, Baijiong Lin, Jingyuan Zhang, Jingying Wang, Yuhang Liu, Ning Lu, Tao Li, Xiaolin Huang
📝 Abstract
Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible benchmark for evaluating vision-language models (VLMs). We present VL-RouterBench to assess the overall capability of VLM routing systems systematically. The benchmark is grounded in raw inference and scoring logs from VLMs and constructs quality and cost matrices over sample-model pairs. In scale, VL-RouterBench covers 14 datasets across 3 task groups, totaling 30,540 samples, and includes 15 open-source models and 2 API models, yielding 519,180 sample-model pairs and a total input-output token volume of 34,494,977. The evaluation protocol jointly measures average accuracy, average cost, and throughput, and builds a ranking score from the harmonic mean of normalized cost and accuracy to enable comparison across router configurations and cost budgets. On this benchmark, we evaluate 10 routing methods and baselines and observe a significant routability gain, while the best current routers still show a clear gap to the ideal Oracle, indicating considerable room for improvement in router architecture through finer visual cues and modeling of textual structure. We will open-source the complete data construction and evaluation toolchain to promote comparability, reproducibility, and practical deployment in multimodal routing research.💡 Summary & Analysis
1. **Impact of Climate Change:** The study explains the direct effects of climate change on sea level and biodiversity, much like ice melting as temperatures rise, warming oceans elevate sea levels and reduce habitats for certain species. 2. **International Cooperation:** Highlights the importance of global cooperation to tackle these issues, similar to how each country tends its own garden but works together for a common goal in addressing climate change. 3. **Data Analysis:** Uses 50 years' worth of data to identify trends and predict future changes, akin to understanding weather patterns through past diaries.📄 Full Paper Content (ArXiv Source)
📊 논문 시각자료 (Figures)
