提出新方法提升视觉语言模型加速时的定位精度
Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
- 采用多阶段剪枝结合早期均匀采样，更好保留图像关键区域
- 在视觉定位任务上性能提升超5倍，计算开销相近
- 适合需要精细视觉理解的下游应用，如目标定位
近期视觉语言模型加速方法通过在语言模型中提前剪枝视觉令牌实现高效压缩，虽在多数任务表现良好，但在以视觉为中心的任务（如定位）中表现异常。我们发现该方法会过度剪除图像上部的令牌，而现有基准测试未能有效识别此缺陷。为此，我们提出FEATHER（Fast and Effective Acceleration wiTH Ensemble cRiteria），通过多阶段剪枝与早期均匀采样确保图像覆盖范围，增强关键令牌保留。在保持相近计算成本下，相较原方法在视觉定位基准上性能提升超过5倍。
原文摘要 · Abstract (English)
Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work, we examine the popular acceleration approach of early pruning of visual tokens inside the language model. Surprisingly, we find that while strong performance is maintained across many tasks, it exhibits drastically different behavior for a subset of vision-centric tasks such as localization. Upon further investigation, we uncover a core issue with the acceleration approach where most tokens towards the top of the image are pruned away. Yet, on many benchmarks aiming to evaluate vision-centric capabilities, strong performance persists with the flawed pruning strategy, highlighting these benchmarks' limited ability to assess fine-grained visual capabilities. Based on these findings, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble cRiteria), a straightforward approach that resolves the discovered early-layer pruning issue and further enhances the preservation of relevant tokens via multistage pruning with early uniform sampling to ensure broad image coverage. With comparable computational savings, we find that FEATHER achieves more than 5x performance improvement on the vision-centric localization benchmarks compared to the original acceleration approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。