提出新方法让视觉标记剪枝更高效,性能损失小、速度提升快。
Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering
- 基于覆盖理论重构剪枝问题,实现双目标平衡
- 仅用11.1%标记保留LLaVA-1.5 96.4%性能,提速1.3-1.5倍
- 适用于多种多模态模型,可推广至视频等复杂任务
现有视觉标记剪枝方法采用静态策略,忽视不同任务中提示对齐与视觉保留的相对重要性,导致性能不稳定。本文首次基于豪斯多夫距离推导出剪枝的闭式误差界,统一刻画两项目标的贡献。结合ε-覆盖理论,揭示二者内在权衡关系,并量化固定预算下的最优达成水平。为此提出多目标平衡覆盖(MoB),将剪枝重构成双目标覆盖问题,通过贪心半径调配实现预算分配。MoB具备可证明的性能边界和线性可扩展性,能适应高难度剪枝场景。大量实验表明,使用仅11.1%原始视觉标记,即可在LLaVA-1.5-7B上保持96.4%性能,并使LLaVA-Next-7B加速1.3–1.5倍,性能损失极小。在Qwen2-VL与Video-LLaVA上的评估也证实,MoB可无缝集成到先进多模态大模型及多样化视觉语言任务中。
原文摘要 · Abstract (English)
Existing visual token pruning methods target prompt alignment and visual preservation with static strategies, overlooking the varying relative importance of these objectives across tasks, which leads to inconsistent performance. To address this, we derive the first closed-form error bound for visual token pruning based on the Hausdorff distance, uniformly characterizing the contributions of both objectives. Moreover, leveraging $ε$-covering theory, we reveal an intrinsic trade-off between these objectives and quantify their optimal attainment levels under a fixed budget. To practically handle this trade-off, we propose Multi-Objective Balanced Covering (MoB), which reformulates visual token pruning as a bi-objective covering problem. In this framework, the attainment trade-off reduces to budget allocation via greedy radius trading. MoB offers a provable performance bound and linear scalability with respect to the number of input visual tokens, enabling adaptation to challenging pruning scenarios. Extensive experiments show that MoB preserves 96.4% of performance for LLaVA-1.5-7B using only 11.1% of the original visual tokens and accelerates LLaVA-Next-7B by 1.3-1.5$\times$ with negligible performance loss. Additionally, evaluations on Qwen2-VL and Video-LLaVA confirm that MoB integrates seamlessly into advanced MLLMs and diverse vision-language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。