用强化学习精准删减视觉令牌,提速15%且不丢精度
GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models

- 基于强化学习构建离散令牌剪枝框架,摆脱梯度近似陷阱
- 在多个基准上实现更优权衡,压缩后推理速度提升最高15%
- 无需重训练即可适配任意压缩率,适合部署优化场景
视觉语言模型处理海量视觉令牌导致计算开销巨大。现有训练感知剪枝方法多依赖连续梯度松弛,但视觉令牌剪枝本质上是离散非凸组合问题,连续近似常陷入次优局部极小值,尤其在激进压缩预算下表现不佳。为此,我们提出GRIP-VLM,一种基于强化学习的组相对重要性剪枝框架。该框架将剪枝建模为马尔可夫决策过程,采用组相对策略优化(GRPO)范式,结合监督预热直接探索离散选择空间。集成预算感知评分器后,轻量级智能体可动态评估每令牌重要性,并自适应任意压缩比而无需重训练。跨多元模态基准的大量实验表明,GRIP-VLM持续优于启发式与监督学习基线,在相同精度下实现更高的帕累托前沿,推理速度最高提升15%。
原文摘要 · Abstract (English)
In Vision-Language Models (VLMs), processing a massive number of visual tokens incurs prohibitive computational overhead. While recent training-aware pruning methods attempt to selectively discard redundant tokens, they largely rely on continuous-gradient relaxations. However, visual token pruning is inherently a discrete, non-convex combinatorial problem; consequently, these continuous approximations frequently trap the optimization in sub-optimal local minima, especially under aggressive compression budgets. To overcome this fundamental bottleneck, we propose GRIP-VLM, a Group-Relative Importance Pruning framework driven by Reinforcement Learning. Rather than relying on smooth-gradient assumptions, GRIP-VLM formulates pruning as a Markov Decision Process, employing a Group Relative Policy Optimization (GRPO) paradigm anchored by supervised warm-up to directly explore the discrete selection space. Integrated with a budget-aware scorer, our lightweight agent dynamically evaluates per-token importance and adapts to arbitrary compression ratios without retraining. Extensive experiments across diverse multimodal benchmarks demonstrate that GRIP-VLM consistently outperforms heuristic and supervised-learning baselines, achieving a superior Pareto frontier and delivering up to a 15\% inference speedup at equal accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。