提出ShaRP框架，让视频大模型推理快5倍且几乎不丢性能。
ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Models
- 通过优化信息聚合、校准位置偏差、减少冗余，提升浅层注意力剪枝可靠性。
- 在86%计算量缩减下保持97.2%原模型性能，预填充阶段提速5.1倍。
- 适合追求高效推理的视频大模型应用，无需重新训练。
视频大语言模型（VLLM）因视觉标记数量庞大，导致预填充阶段计算开销巨大。尽管基于注意力的标记剪枝是潜在加速策略，但在浅层解码器中高压缩比常引发严重性能下降。本文揭示了浅层注意力剪枝被忽视的失效机制：早期解码层注意力分数难以准确反映标记重要性，导致高压缩下选择不稳定。该现象源于交互不足、内容无关的位置偏置以及高注意力标记间的冗余，共同扭曲了注意力重要性估计。为此，我们提出ShaRP统一剪枝框架，通过增强局部信息聚合、校准位置偏置、降低冗余，恢复可靠的注意力标记选择。大量实验表明，ShaRP在降低86% TFLOPs的同时，保留约97.2%的原始性能，预填充阶段实现5.1倍加速，为无训练高效推理提供了可扩展解决方案。
原文摘要 · Abstract (English)
Video Large Language Models (VLLMs) incur substantial prefilling cost due to the large number of visual tokens. While attention-based token pruning offers a promising acceleration strategy, applying it at shallow decoder layers often causes severe performance degradation under high compression ratios, limiting its practical benefits. In this work, we uncover an overlooked failure mode in shallow-layer attention pruning: attention scores in early decoder layers can become unreliable indicators of token utility, resulting in unstable token selection under aggressive compression. We show that this effect arises from the joint influence of insufficient token interaction, content-agnostic positional bias, and redundancy among high-attention tokens, which together distort attention-based importance estimation before informative representations fully emerge. Motivated by this insight, we propose ShaRP, a unified pruning framework that restores reliable attention-based token selection by jointly improving local information aggregation, calibrating positional bias, and reducing redundancy. Extensive evaluations show that ShaRP preserves about 97.2% of the original performance while reducing TFLOPs by 86% and achieving a 5.1x speedup in the prefilling stage, providing a scalable solution for efficient training-free VLLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。