PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective

Recently, reducing redundant visual tokens in vision-language models (VLMs) to accelerate VLM inference has emerged as a hot topic. However, most existing methods rely on heuristics constructed based on inter-visual-token similarity or cross-modal visual-text similarity, which gives rise to certain limitations in compression performance and practical deployment. In contrast, we propose PIO-FVLM from the perspective of inference objectives, which transforms visual token compression into preserving output result invariance and selects tokens primarily by their importance to this goal. Specially, vision tokens are reordered with the guidance of token-level gradient saliency generated by our designed layer-local proxy loss, a coarse constraint from the current layer to the final result. Then the most valuable vision tokens are selected following the non-maximum suppression (NMS) principle. The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment. It can be deployed independently as an encoder-free method, or combined with encoder compression approaches like VisionZip for use as an encoder-involved method. On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance, with a 2.67$\times$ prefill speedup, 2.11$\times$ inference speedup, 6.22$\times$ lower FLOPs, and 6.05$\times$ reduced KV Cache overhead. Our code is available at https://github.com/ocy1/PIO-FVLM.

💡 Research Summary

Vision‑language models (VLMs) have achieved impressive multimodal reasoning capabilities, but the large number of visual tokens generated by the vision encoder (often several hundred) creates a significant computational and memory burden during inference. Existing token‑reduction methods fall into two categories. Vision‑Encoder‑Involved (w/ VE) approaches compress tokens inside the vision encoder (e.g., ToMe, VisionZip, HoloV, SCOPE), while Vision‑Encoder‑Free (w/o VE) approaches leave the encoder output untouched and prune tokens directly within the language model (e.g., FastV, SparseVLM, PyramidDrop). Both families rely on heuristics: either similarity among visual tokens or cross‑modal attention scores. These heuristics suffer from two major drawbacks. First, attention‑map‑based methods are incompatible with FlashAttention, limiting real‑world speed‑up. Second, similarity‑based methods can misjudge token importance because they do not directly consider the final inference objective, leading to degraded performance in certain queries.

PIO‑FVLM introduces a fundamentally different perspective: instead of heuristics, it treats token reduction as an objective‑preserving problem. The goal is to keep the model’s final output (e.g., the generated answer) as unchanged as possible after removing visual tokens. To achieve this without any additional training, the authors propose a layer‑local proxy loss (L_l) that approximates the influence of a given layer’s features on the final prediction. Concretely, for a pruning layer (l), the hidden states (H_l) are fed through the original prediction head of the LLM to obtain soft logits (p_{lt}) and pseudo hard labels (\hat y_{lt}= \arg\max p_{lt}). A cross‑entropy loss is then computed over a small tail window of tokens (the last (K_{pos}) positions), yielding (L_l = \frac{1}{|P_l|}\sum_{t\in P_l} \text{CE}(p_{lt},\hat y_{lt})). This loss does not require ground‑truth labels; the pseudo‑labels simply enforce that the current output be treated as correct.

The gradient of (L_l) with respect to the input of the pruning layer, (\partial L_l / \partial H_{l-1}), provides a saliency score for each visual token: (s_{li}= | \partial L_l / \partial H_{l-1,i} |_2). A large saliency indicates that perturbing the token would significantly affect the proxy loss, and therefore the final output. Tokens with high saliency are candidates for retention.

However, naïvely keeping the top‑K tokens by saliency tends to select spatially clustered tokens, reducing global visual coverage. To address this, PIO‑FVLM applies a non‑maximum suppression (NMS)‑style selection on the saliency‑ranked list. Token features (v_i) (the hidden states) are L2‑normalized to (u_i). An upper‑triangular similarity matrix (S_{ij}= \langle u_i, u_j\rangle) is computed. Tokens are processed in descending saliency order; a token is kept only if its similarity to every already‑selected token is below a predefined threshold (e.g., 0.7). This procedure preserves important tokens while explicitly discouraging redundancy, ensuring a diverse set of visual cues.

The overall compression pipeline operates during the pre‑fill stage of the LLM. Several pruning layers are chosen from shallow to deep, and at each layer the following steps are performed: (1) compute the layer‑local proxy loss, (2) back‑propagate to obtain saliency scores, (3) apply NMS to select a reduced set of tokens. The number of retained tokens decreases progressively, allowing early layers to prune only a few tokens (high fault tolerance) and deeper layers to prune more aggressively as the proxy loss becomes a better approximation of the final loss.

A key advantage of this design is its compatibility with FlashAttention and other optimized attention kernels. The method does not modify the internal attention computation; it merely adds a forward pass through the prediction head and a single‑layer backward pass, both of which are supported by standard autograd pipelines. Consequently, PIO‑FVLM can be deployed as a plug‑and‑play module without any changes to the underlying model code or hardware.

Extensive experiments were conducted on three popular VLMs (LLaVA‑Next‑7B, LLaVA‑1.5‑7B, and a third unspecified model) across eight benchmarks (GQA, MMB, MMB‑cn, MME, POPE, SQA, VQA‑v2, TextVQA). Token budgets of 33 %, 22 %, and 11 % of the original visual tokens were evaluated. With only 11.1 % of tokens retained (≈64 tokens), PIO‑FVLM achieved an average of 94.7 % of the original performance, outperforming state‑of‑the‑art methods such as FastV, PDrop, SparseVLM, DART, VisionZip, HoloV, and SCOPE. FLOPs were reduced by 6.22×, KV‑cache memory by 6.05×, pre‑fill latency improved by 2.67×, and overall inference speed by 2.11×. Moreover, when combined with VisionZip (an encoder‑involved compression technique), PIO‑FVLM retained its superiority, demonstrating flexibility as both an encoder‑free and encoder‑involved solution.

In summary, PIO‑FVLM introduces three novel components: (1) a layer‑local proxy loss that approximates the final inference objective without requiring ground‑truth labels, (2) gradient‑based saliency scores that directly measure token importance for output preservation, and (3) an NMS‑based selection mechanism that balances importance and diversity. The method is training‑free, highly efficient, and fully compatible with modern attention optimizations, offering a practical pathway to accelerate VLM inference while maintaining near‑original accuracy.