发现并利用视觉令牌冗余，让解码器型多模态模型更快更省资源。
RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs
- 通过动态前馈网络与空注意力机制，无训练分析视觉令牌计算冗余。
- 实验显示解码器架构存在显著结构化冗余，可大幅降低计算量。
- 无需重训练即可加速推理，适合追求高效部署的开发者使用。
当前多模态大模型架构在性能与效率间面临关键权衡：解码器型架构性能更高但效率低，交叉注意力型架构效率高但性能弱。根本差异在于视觉令牌的处理方式：解码器型对视觉令牌执行自注意力与前馈网络（FFN）操作，而交叉注意力型跳过这些计算。为探究此类高开销计算是否存在冗余，我们提出一种无需训练的分析框架。该框架包含探针激活动态前馈网络与空注意力机制，可灵活减少视觉令牌的计算量，并结合层排序算法确定优先降级的层。大量实验表明，解码器型模型存在显著、结构化且聚类的冗余，为未来架构设计提供重要启示。此外，将此减法框架作为无训练推理加速手段，可在保持或超越现有先进方法性能的同时，与现有技术兼容。代码将公开于 https://github.com/L-Hugh/RedundancyLens。
原文摘要 · Abstract (English)
Current Multimodal Large Language Model (MLLM) architectures face a critical tradeoff between performance and efficiency: decoder-only architectures achieve higher performance but lower efficiency, while cross-attention-based architectures offer greater efficiency but lower performance. The key distinction lies in how visual tokens are processed. Decoder-only architectures apply self-attention and FFN operations on visual tokens, while cross-attention architectures skip these computations. To investigate whether redundancy exists in this computationally expensive process, we propose a training-free framework for analyzing trained MLLMs. It consists of Probe-Activated Dynamic FFN and Hollow Attention, which enable adjustable reductions in computations for visual tokens, as well as a Layer Ranking Algorithm that prioritizes layers for these reductions. Extensive experiments demonstrate substantial, structured, and clustered redundancy unique to decoder-only MLLMs, offering valuable insights for future MLLM architecture design. Furthermore, by leveraging our reduction framework as a training-free inference acceleration approach, we achieve performance comparable to or better than state-of-the-art methods while remaining compatible with them. Code will be publicly available at https://github.com/L-Hugh/RedundancyLens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。