修复视觉令牌压缩中的位置与注意力失真，提升多模态大模型推理效率
Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
- 通过相对距离校准注意力权重，恢复压缩后丢失的视觉关注
- 设计锚点选择机制，减少特征平均导致的信息损失
- 在多个基准上超越现有方法，兼顾精度与计算效率
多模态大语言模型在视觉-语言任务中取得显著进展，但海量视觉令牌带来的二次计算复杂度导致内存和延迟瓶颈。尽管已有视觉令牌压缩（VTR）策略缓解该问题，但现有方法忽略完整序列与压缩序列间的位置和注意力一致性，造成表征失真。为此，我们提出RESTORE，一种新型VTR框架，在保持高效的同时修正位置与注意力失真。具体而言，提出一种简单有效的校准方法，基于相对距离增强注意力权重以恢复丢失的视觉关注；同时引入独特的锚点选择机制，减轻特征平均过程中的信息损失。在多个基准上的实验表明，该方法持续提升各类压缩方法的准确性，达到当前最优性能，且保持计算效率。项目页面见：https://cvlab.yonsei.ac.kr/projects/RESTORE
原文摘要 · Abstract (English)
Recent advancements in Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision-language tasks, yet the quadratic computational complexity arising from the vast number of visual tokens incurs significant memory and latency bottlenecks. While visual token reduction (VTR) strategies have been explored to mitigate this burden, existing methods overlook the positional and attentional consistency between the full and reduced sequences, resulting in a distorted representation. To this end, we propose RESTORE, a novel VTR framework that rectifies the positional and attentional distortions while maintaining efficiency. Specifically, we present a simple yet effective calibration method that restores lost visual attention by augmenting attention weights based on relative distances. We also introduce a distinctive anchor selection for token merging to mitigate information loss during feature averaging. Experimental results on multiple benchmarks demonstrate that our method consistently improves the accuracy of various reduction methods, achieving state-of-the-art performance while maintaining computational efficiency. Project page is available at https://cvlab.yonsei.ac.kr/projects/RESTORE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。