通过验证器引导的分阶段剪枝，让视频大模型推理快2.68倍且不丢精度。
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
- 用验证器信号选出关键视频帧，再均匀剪掉冗余帧，实现无损加速。
- 在四个数据集上实测，最高提速2.68倍，比原模型更稳定高效。
- 无需重新训练，适配主流视频大模型，适合追求推理速度的研究者。
视频大语言模型（Vid-LLMs）在理解视频内容方面表现出强大能力，但其依赖密集的视频标记表示，导致预填充和解码阶段存在显著的内存与计算开销。为缓解现有视频标记压缩方法的信息损失并无损加速解码过程，我们提出SpecVLM——一种面向视频大模型的免训练推测解码框架，采用分阶段视频标记剪枝策略。基于新发现：草案模型的推测对视频标记剪枝敏感度低，SpecVLM可剪除高达90%的视频标记，实现高效推测而不牺牲准确性。该方法包含两阶段剪枝：第一阶段由验证器（目标模型）的注意力信号指导选择高信息量标记；第二阶段以空间均匀方式剪除剩余冗余标记。在四个视频理解基准上的广泛实验表明，SpecVLM有效且鲁棒，使LLaVA-OneVision-72B的解码速度提升达2.68倍，Qwen2.5-VL-32B提升2.11倍。代码已开源于https://github.com/zju-jiyicheng/SpecVLM。
原文摘要 · Abstract (English)
Video large language models (Vid-LLMs) have shown strong capabilities in understanding video content. However, their reliance on dense video token representations introduces substantial memory and computational overhead in both prefilling and decoding. To mitigate the information loss of recent video token reduction methods and accelerate the decoding stage of Vid-LLMs losslessly, we introduce SpecVLM, a training-free speculative decoding (SD) framework tailored for Vid-LLMs that incorporates staged video token pruning. Building on our novel finding that the draft model's speculation exhibits low sensitivity to video token pruning, SpecVLM prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. To achieve this, we performs a two-stage pruning process: Stage I selects highly informative tokens guided by attention signals from the verifier (target model), while Stage II prunes remaining redundant ones in a spatially uniform manner. Extensive experiments on four video understanding benchmarks demonstrate the effectiveness and robustness of SpecVLM, which achieves up to 2.68$\times$ decoding speedup for LLaVA-OneVision-72B and 2.11$\times$ speedup for Qwen2.5-VL-32B. Code is available at https://github.com/zju-jiyicheng/SpecVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。