arXiv:2604.20937cs.LG2026-04

针对视频大模型细粒度理解难题,提出识别并剔除干扰注意力的无用视觉标记。

Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs

论文配图:Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
图 1 · 摘自论文原文
  • 通过分析发现冗余视觉标记会误导模型判断,影响精准理解
  • 新方法在剪除90%标记后仍显著提升幻觉检测等任务表现
  • 可无缝接入现有剪枝方法,适合追求高效高精度视频理解的场景

视频大语言模型因输入大量视觉标记导致推理延迟过高。为降低计算开销,无需训练的视觉标记剪枝方法应运而生;然而现有方法主要在多选题问答(MCQA)基准上验证,此类任务常依赖粗粒度线索。本研究揭示:这些方法在需要精确视觉定位的细粒度理解任务(如幻觉评估)中性能急剧下降。我们系统分析发现,‘sink tokens’——即语义无信息且过度吸引注意力的标记——是主要障碍。当这些标记未被剔除时,会扭曲模型对视觉证据的判断。基于此,我们提出Sink-Token-aware Pruning(SToP),一种简单有效的即插即用方法:引入sink分数量化每个标记成为干扰源的可能性,并将其融入现有时空剪枝方法以抑制其留存,从而提升视频理解能力。我们在多个基准(包括幻觉、开放式生成、组合推理和MCQA)上测试SToP,应用于VisionZip、FastVid和Holitom等先进剪枝方法,结果表明:即使剪除高达90%的视觉标记,性能仍显著提升。

原文摘要 · Abstract (English)

Video Large Language Models (Video LLMs) incur high inference latency due to a large number of visual tokens provided to LLMs. To address this, training-free visual token pruning has emerged as a solution to reduce computational costs; however, existing methods are primarily validated on Multiple-Choice Question Answering (MCQA) benchmarks, where coarse-grained cues often suffice. In this work, we reveal that these methods suffer a sharp performance collapse on fine-grained understanding tasks requiring precise visual grounding, such as hallucination evaluation. To explore this gap, we conduct a systematic analysis and identify sink tokens--semantically uninformative tokens that attract excessive attention--as a key obstacle to fine-grained video understanding. When these sink tokens survive pruning, they distort the model's visual evidence and hinder fine-grained understanding. Motivated by these insights, we propose Sink-Token-aware Pruning (SToP), a simple yet effective plug-and-play method that introduces a sink score to quantify each token's tendency to behave as a sink and applies this score to existing spatial and temporal pruning methods to suppress them, thereby enhancing video understanding. To validate the effectiveness of SToP, we apply it to state-of-the-art pruning methods (VisionZip, FastVid, and Holitom) and evaluate it across diverse benchmarks covering hallucination, open-ended generation, compositional reasoning, and MCQA. Our results demonstrate that SToP significantly boosts performance, even when pruning up to 90% of visual tokens.

视频理解剪枝大模型优化细粒度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。