通过轻量化视觉令牌计算，提升多模态模型效率且不损失性能
Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM
- 用代理令牌替代原始视觉令牌的复杂计算
- 在保持性能的同时减少计算开销，部分场景还提升效果
- 可与去冗余令牌方法结合，进一步提升效率
大型多模态模型在多模态任务中表现优异，但因对视觉令牌进行过度计算而面临显著算力挑战。不同于仅关注令牌层面冗余的方法，我们识别并研究了视觉令牌在计算层级上的冗余，确保无信息损失。关键洞察是：预训练视觉编码器输出的视觉令牌无需在解码器仅有的多模态模型中执行全部重计算操作（如自注意力、前馈网络），通过合理设计可轻量化处理。我们设计了一系列实验，逐步发现并压缩视觉相关计算冗余。基于此，提出新方法ProxyV，利用代理视觉令牌缓解原视觉令牌的计算负担。ProxyV在不牺牲性能的前提下显著提升效率，甚至在适度效率提升的场景中带来明显性能增益。此外，其灵活性体现在可与令牌压缩方法结合，进一步增强效率。代码将公开于 https://github.com/penghao-wu/ProxyV。
原文摘要 · Abstract (English)
Large multimodal models excel in multimodal tasks but face significant computational challenges due to excessive computation on visual tokens. Unlike token reduction methods that focus on token-level redundancy, we identify and study the computation-level redundancy on vision tokens to ensure no information loss. Our key insight is that vision tokens from the pretrained vision encoder do not necessarily require all the heavy operations (e.g., self-attention, FFNs) in decoder-only LMMs and could be processed more lightly with proper designs. We designed a series of experiments to discover and progressively squeeze out the vision-related computation redundancy. Based on our findings, we propose ProxyV, a novel approach that utilizes proxy vision tokens to alleviate the computational burden on original vision tokens. ProxyV enhances efficiency without compromising performance and can even yield notable performance gains in scenarios with more moderate efficiency improvements. Furthermore, the flexibility of ProxyV is demonstrated through its combination with token reduction methods to boost efficiency further. The code will be made public at this https://github.com/penghao-wu/ProxyV URL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。