Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMs

Vision capabilities in vision large language models (VLLMs) have consistently lagged behind their linguistic capabilities. In particular, numerous benchmark studies have demonstrated that VLLMs struggle when fine-grained visual information or spatial reasoning is required. However, we do not yet understand exactly why VLLMs struggle so much with these tasks relative to others. Some works have focused on visual redundancy as an explanation, where high-level visual information is uniformly spread across numerous tokens and specific, fine-grained visual information is discarded. In this work, we investigate this premise in greater detail, seeking to better understand exactly how various types of visual information are processed by the model and what types of visual information are discarded. To do so, we introduce a simple synthetic benchmark dataset that is specifically constructed to probe various visual features, along with a set of metrics for measuring visual redundancy, allowing us to better understand the nuances of their relationship. Then, we explore fine-tuning VLLMs on a number of complex visual tasks to better understand how redundancy and compression change based upon the complexity of the data that a model is trained on. We find that there is a connection between task complexity and visual compression, implying that having a sufficient ratio of high complexity visual data is crucial for altering the way that VLLMs distribute their visual representation and consequently improving their performance on complex visual tasks. We hope that this work will provide valuable insights for training the next generation of VLLMs.

💡 Research Summary

The paper investigates why vision‑large language models (VLLMs) consistently underperform on fine‑grained visual tasks such as object grounding and spatial reasoning, despite strong linguistic capabilities. The authors hypothesize that visual redundancy—where high‑level information is spread uniformly across many visual tokens and detailed information is lost—plays a central role, but they seek a deeper, quantitative understanding.

To this end, they introduce a suite of compression metrics. Token‑norm based measures include the Gini coefficient (inequality of L2 norms across token embeddings), normalized entropy (information content of the norm distribution), and coefficient of variation (relative variability). Rank‑based measures are derived from singular‑value decomposition (SVD) of token matrices: stable rank (effective dimensionality), participation ratio (number of singular values contributing significantly), and exponential entropy (Shannon entropy of singular values). These metrics capture both how evenly information is distributed and how many latent dimensions are actually used.

A synthetic benchmark is constructed using 2‑D shapes on a white background. By systematically varying the number of objects, shape types, colors, sizes, and color combinations, the dataset spans a continuum from low to high visual complexity. This controlled setting allows the authors to directly correlate task complexity with compression metrics.

Zero‑shot experiments are performed on two state‑of‑the‑art VLLMs—Molmo and Llama 3.2—representing different multimodal architectures (joint token concatenation vs. cross‑attention). The authors compute the compression metrics across all transformer layers and train linear probes on each token position to predict specific visual attributes (e.g., most common object, largest object). Results show that, for both models, visual information is highly redundant: most tokens encode a global view of the image rather than specialized details. As visual complexity increases, Gini and stable rank rise, indicating that information becomes more evenly spread across tokens, while low‑complexity images exhibit more concentrated information.

Spearman correlation analyses reveal that higher redundancy (higher Gini, stable rank, participation ratio) strongly predicts poorer performance on complex downstream tasks such as object counting and relational reasoning. This establishes a concrete link between redundancy and failure modes.

The authors then explore fine‑tuning effects. They fine‑tune each VLLM on two families of tasks: (1) grounding tasks that require linking specific objects to textual mentions, and (2) spatial‑reasoning tasks that demand relational judgments. After fine‑tuning, the compression metrics are recomputed. Grounding fine‑tuning markedly increases norm‑based inequality (higher Gini) and reduces stable rank, suggesting that certain visual tokens become dedicated carriers of object‑specific information. Spatial‑reasoning fine‑tuning, by contrast, leaves the token‑norm distribution largely unchanged but induces larger changes in the text‑side representations, implying that the model solves these tasks by leveraging textual context rather than reorganizing visual token usage.

Ablation experiments, where a random fraction of visual tokens is dropped, confirm that models with higher redundancy are more robust to token loss up to a point, but once a critical ablation ratio is crossed, performance collapses sharply. The critical point aligns with layers that exhibit low stable rank and participation ratio, reinforcing the idea that low‑dimensional token spaces are fragile.

Overall, the paper makes three major contributions: (1) a comprehensive set of quantitative metrics for visual redundancy that go beyond prior token‑pruning studies; (2) empirical evidence that visual redundancy is tightly coupled with task complexity, explaining why VLLMs fail on fine‑grained tasks; and (3) insight that fine‑tuning on sufficiently complex visual data can reshape token utilization, but the nature of the task (grounding vs. spatial reasoning) determines whether visual or textual representations are primarily altered.

These findings suggest that future VLLM training should deliberately incorporate high‑complexity visual examples and consider architectural adjustments that allow for more specialized visual token pathways. The proposed metrics and synthetic benchmark provide practical tools for diagnosing and mitigating redundancy, paving the way toward VLLMs that can match human‑level visual reasoning across a broader spectrum of tasks.