通过强化学习提升多模态大模型的令牌稀疏性,大幅降低计算开销。
Sparsity Forcing: Reinforcing Token Sparsity of MLLMs
- 设计基于强化学习的后训练框架,动态优化不同预算下的令牌保留比例。
- 在13个图像与视频基准上将令牌减少比例从20%提升至75%,准确率下降极小。
- 适合追求高效推理、需处理长上下文多模态任务的研究者与开发者。
稀疏注意力机制通过有选择地处理关键令牌来降低计算开销,同时保持最小精度损失。然而,现有方法大多仅利用模型固有的稀疏性,导致在约50%令牌减少时达到瓶颈,难以进一步压缩预算而不损害性能。其他方法尝试通过可训练稀疏注意力或锐化诱导正则项强制稀疏性,但前者固定模式忽略输入与层间动态,后者优化代理目标而无法直接控制令牌预算。本文提出一种名为‘Sparsity Forcing’的简单强化学习后训练框架,显式强化多模态大模型(MLLMs)的令牌稀疏性。该方法通过多轮采样探索效率-精度权衡,将令牌减少率与答案正确性共同作为联合奖励。通过组内对比,更高效且正确的回答获奖励,低效或错误回答受惩罚,从而将令牌节省转化为端到端、推理一致的优化目标。在十三个图像与视频基准上,Sparsity Forcing 将 Qwen2-VL/Qwen2.5-VL 的令牌减少比例从20%提升至75%,准确率下降微小,长上下文推理内存最高减少3倍,解码速度最快提升3.3倍。
原文摘要 · Abstract (English)
Sparse attention mechanisms aim to reduce computational overhead with minimal accuracy loss by selectively processing salient tokens. Despite their effectiveness, most methods merely exploit a model's inherent sparsity and thus plateau at moderate budgets (about 50\% token reduction), with little headroom to push budget lower without hurting accuracy. Other approaches attempt to enforce sparsity through trainable sparse attention or sharpness-inducing regularizers, but these either fix rigid patterns that ignore input and layer dynamics, or optimize proxy objectives without direct control over token budgets. In this paper, we explicitly reinforce token sparsity in well-posed multimodal large language models (MLLMs) through a simple RL-based post-training framework named \textit{Sparsity Forcing}. Our method explores the efficiency-accuracy trade-off by running multiple rollouts with different token budgets, where both efficiency (token reduction ratio) and performance (answer correctness) are formulated as joint rewards. By contrasting rollouts within each group, the more efficient and correct answer is rewarded while less efficient or incorrect ones are penalized, thereby turning token saving into an end-to-end, inference-consistent optimization objective. Across thirteen image and video benchmarks, Sparsity Forcing raises token reduction ratio on Qwen2-VL/Qwen2.5-VL from 20\% to 75\% with minimal accuracy decline, significantly reducing long-context inference memory by up to 3$\times$ while speeding up decoding by up to 3.3$\times$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。