v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning

When thinking with images, humans rarely rely on a single glance: they revisit visual evidence while reasoning. In contrast, most Multimodal Language Models encode an image once to key-value cache and then reason purely in text, making it hard to re-ground intermediate steps. We empirically confirm this: as reasoning chains lengthen, models progressively lose focus on relevant regions. We introduce v1, a lightweight extension for active visual referencing via point-and-copy: the model selects relevant image patches and copies their embeddings back into the reasoning stream. Crucially, our point-and-copy mechanism retrieves patches using their semantic representations as keys, ensuring perceptual evidence remains aligned with the reasoning space. To train this behavior, we build v1, a dataset of 300K multimodal reasoning traces with interleaved grounding annotations. Across multimodal mathematical reasoning benchmarks, v1 consistently outperforms comparable baselines. We plan to release the model checkpoint and data.

💡 Research Summary

The paper addresses a fundamental limitation of current multimodal large language models (MLLMs): they encode an image only once into a key‑value cache and then perform all reasoning purely in the textual domain. This design leads to “visual grounding decay,” where attention to image tokens and especially to task‑relevant regions steadily diminishes as the generation proceeds, a problem that becomes acute in long‑chain reasoning tasks such as multimodal mathematical problem solving.

To mitigate this, the authors propose v1, a lightweight extension that equips an MLLM with a “point‑and‑copy” mechanism. At each decoding step the model simultaneously produces (1) a standard vocabulary distribution and (2) a pointing distribution over the positions of the input image patches. The pointing head computes logits by attending the decoder hidden state to the continuous image embeddings via learned linear projections, similar to standard attention scoring. The two logits are concatenated, forming an augmented output space (\bar V = V \cup C), where (C) denotes the set of image‑patch embeddings. When the model selects an index in (C), the corresponding patch embedding is copied and injected as the next token’s input, allowing the model to re‑access visual evidence dynamically during generation. No gating scalar is required because the vocabularies are disjoint.

Training this behavior requires fine‑grained supervision linking each reasoning step to a specific visual region. The authors therefore construct v1g, a dataset of 300 K multimodal mathematical reasoning traces with interleaved grounding annotations. The pipeline consists of (1) oversampling reasoning paths from a pretrained MLLM (the TVC training set), (2) using a strong LLM (Gemini‑2.0‑flash) to decompose each path into explicit visual queries expressed as “detect” calls, and (3) aligning each query with a bounding box in the original image. The resulting data provide token‑level pointers to image patches that can be used to train the pointing head.

Empirical evaluation on three established multimodal math benchmarks—MathVista, MathVision, and MathVerse—shows that v1 consistently outperforms comparable‑size baselines (e.g., TVC‑7B, LLaVA‑CoT) and narrows the gap to much larger models. The gains are especially pronounced on geometry and diagram‑heavy problems where intermediate steps benefit from revisiting specific visual cues. Ablation studies confirm that removing the pointing head or disabling copy‑injection degrades performance markedly, underscoring the importance of dynamic visual access. Moreover, the additional parameters are limited to lightweight linear heads, incurring negligible computational overhead.

The paper also provides a detailed analysis of visual grounding decay using RefCOCO, demonstrating that both total attention to image tokens and the relative attention to target regions decline as decoding progresses. This quantitative evidence motivates the need for mechanisms like v1 that can refresh visual context on demand.

In discussion, the authors note that while patch‑level pointing is effective, finer‑grained coordinate prediction or handling of composite visual queries could further improve performance. They also acknowledge that the automated generation of v1g may inherit occasional LLM errors, suggesting future work on human‑in‑the‑loop verification.

In conclusion, v1 introduces a simple yet powerful pointer‑generator style extension to multimodal language models, enabling them to “look back” at image evidence throughout multi‑step reasoning. The release of the model checkpoint and the large‑scale v1g dataset promises to catalyze further research on grounded multimodal reasoning, tool‑use integration, and more complex vision‑language tasks.