스트리밍 집중 아키텍처를 통한 비전‑언어 모델 가속화

2026 IEEE International Symposium on High-Performance Computer Architecture (HPCA) Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models Chiyue Wei†∗, Cong Guo†∗§, Junyao Zhang†, Haoxuan Shan†, Yifan Xu†, Ziyue Zhang†, Yudong Liu†, Qinsi Wang†, Changchun Zhou†, Hai “Helen” Li†, Yiran Chen† †Duke University ∗Equal contribution §Corresponding author {chiyue.wei, cong.guo, hai.li, yiran.chen}@duke.edu Abstract—Vision-Language Models (VLMs) have demon- strated strong performance on tasks such as video captioning and visual question answering. However, their growing scale and video-level inputs lead to significant computational and memory overhead, posing challenges for real-time deployment on hardware accelerators. While prior work attempts to reduce re- dundancy via token pruning or merging, these methods typically operate at coarse granularity and incur high runtime overhead due to global token-level operations. In this study, we propose Focus, a Streaming Concentration Ar- chitecture that efficiently accelerates VLM inference through pro- gressive, fine-grained redundancy elimination. Focus introduces a multilevel concentration paradigm that hierarchically compresses vision-language inputs at three levels: (1) semantic-guided token pruning based on textual prompts, (2) spatial-temporal block- level concentration using localized comparisons, and (3) vector- level redundancy removal via motion-aware matching. All con- centration steps are tightly co-designed with the architecture to support streaming-friendly, on-chip execution. Focus leverages GEMM tiling, convolution-style layout, and cross-modal attention to minimize off-chip access while enabling high throughput. Im- plemented as a modular unit within a systolic-array accelerator, Focus achieves 2.4× speedup and 3.3× reduction in energy, significantly outperforming state-of-the-art accelerator in both performance and energy efficiency. Full-stack implementation of Focus is open-sourced at https://github.com/dubcyfor3/Focus. I. INTRODUCTION Vision-Language Models (VLMs) [35], [41] have emerged as a cornerstone of multimodal AI, enabling joint reasoning over visual and textual data. By integrating advances from computer vision and natural language processing, VLMs excel at tasks such as video captioning [67], [74], visual question answering [13], [55], and cross-modal retrieval [36]. Following a similar trajectory to Large Language Models (LLMs) [6], [15], modern VLMs have rapidly scaled in size and data, resulting in notable accuracy gains. However, this scaling significantly increases compute and memory demands, posing challenges for deployment, especially on edge devices [53]. Fortunately, video-based inputs offer a key opportunity: high visual redundancy [7], [28], [52], [62], [69]. As shown in Fig. 1(a), adjacent frames often share similar backgrounds and foreground objects. Since VLMs tokenize each frame inde- pendently [35], [74], many tokens across or within frames are redundant. This has motivated techniques such as token prun- ing [50], [62] and token merging [4] to reduce computation. However, most prior work focuses on algorithmic strategies without considering hardware alignment. For instance, Token Merging [4] introduces a ToMe module that increases runtime by up to 36.8% [70]. Recent designs such as AdapTiV [70] and CMC [56] address these inefficiencies at the hardware level. AdapTiV im- plements a simplified ToMe module in hardware, while CMC leverages video-codec-inspired compression (e.g., H.264 [65]) via an external codec block. However, both approaches largely translate existing algorithms without embracing full hardware- algorithm co-design. First, both targeted for Vision Trans- formers (ViTs) [17], focus only on visual redundancy and overlook the cross-modal nature of VLMs. CMC’s codec ignores language inputs, and AdapTiV only supports static images, missing video-language interactions. Second, both operate at global token-level granularity, which is inefficient for both algorithm and hardware due to high overhead and poor locality. To enable efficient VLM deployment, a more holistic co-design approach is needed, one that leverages cross-modal redundancy while aligning with hardware-friendly processing granularity. In this study, we propose a novel architecture, Focus, to accelerate VLM inference by performing streaming con- centration, a multilevel compression technique that removes visual and cross-modal redundancy in a streaming-friendly, on-chip processing fashion. From the algorithmic perspective, Focus performs redun- dancy concentration at three levels of granularity. First, it leverages semantic understanding to retain only visual regions relevant to the textual prompt. Prior work [4], [49], [56], [70] relies on static metrics like token magnitude, which fail to capture prompt-conditioned semantics in VLMs. As shown in Fig. 1(a), attention may shift from a foreground object (e.g., a dog) to a backgr

…(Full text truncated)…

This content is AI-processed based on ArXiv data.