arXiv:2609.05916cs.CVcs.AI2026-09

提出分阶段自适应剪枝方法,高效压缩视觉语言模型的视觉令牌。

STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models

论文配图:STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models
图 1 · 摘自论文原文
  • 分两阶段剪枝:先保留丰富视觉特征,再逐步精炼冗余令牌。
  • 在LLaVA-Video-7B上减少90.5%视觉令牌,性能损失仅7.3%,推理提速2.24倍。
  • 无需训练,适用于多种视觉语言模型,适合部署优化场景。

大型视觉语言模型(LVLMs)虽具备强大的多模态理解能力,但其处理数百至数千个视觉令牌带来显著计算开销。本文针对无需训练的视觉令牌剪枝展开两项互补分析:首先,测量跨模态融合前保留令牌的特征空间覆盖度,发现激进剪枝会丢弃大量视觉信息;其次,追踪解码器各层中从文本到视觉的注意力变化,发现重要视觉令牌随深度显著变化,导致一次性剪枝决策不可靠。因此,有效剪枝应先在融合前保持广泛视觉覆盖,再随跨模态证据演进逐步精炼保留令牌。为此,我们提出训练免费的两阶段框架STAR-Pro(STage-Wise Adaptive Token Reduction with Progressive Refinement)。其自适应阶段利用带枢轴的QR分解构建超预算的特征覆盖候选集;其渐进阶段在选定解码器层利用动态演变的文本-视觉注意力,在目标层平均令牌预算下剪枝嵌套幸存集。在涵盖多种架构的七种LVLM及18个图像与视频基准上的大量实验表明,STAR-Pro在激进剪枝下仍具有效性。在LLaVA-Video-7B上,视觉令牌减少90.5%,保留基线性能的92.7%,实测推理速度提升2.24倍。代码已开源。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose substantial computational overhead, motivating training-free visual token pruning. In this work, we conduct two complementary analyses of visual token pruning. First, we measure the feature-space coverage of tokens retained before cross-modal fusion and find that aggressive pruning discards substantial visual information. Second, we track text-to-visual attention across decoder layers and find that the visual tokens considered important change substantially with depth, making one-shot pruning decisions unreliable. Together, these findings show that effective pruning should preserve broad visual coverage before fusion and progressively refine the retained tokens as cross-modal evidence evolves during fusion. We therefore propose STAR-Pro (STage-Wise Adaptive Token Reduction with Progressive Refinement), a training-free two-stage framework. Its Adaptive Stage applies pivoted QR to construct an over-budget feature-coverage candidate pool, while its Progressive Stage uses evolving text-to-visual attention at selected decoder layers to prune a nested survivor set under a target layer-average token budget. Extensive experiments across seven LVLMs spanning multiple architectures and 18 image and video benchmarks demonstrate the effectiveness of STAR-Pro under aggressive pruning. On LLaVA-Video-7B, STAR-Pro reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a $2.24\times$ measured inference speedup. Code is available at https://github.com/EasonAI-5589/starpro.

视觉语言模型令牌剪枝高效推理无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。