不训练即可加速多模态大模型，删冗余、补信息、保性能。
Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration
- 用视觉编码器中的冗余度度量筛选并删除无用视觉标记。
- 通过相关性机制让保留标记回收被删标记的信息，减少内容损失。
- 无需重训，适配多种模型和任务，最高降14.7倍计算量。
多模态大语言模型（MLLM）的计算复杂度随上下文长度呈二次增长，带来显著的计算与内存挑战，阻碍其实际部署。本文提出一种「筛选-关联-压缩」框架，通过系统优化预填充阶段的多模态上下文长度来加速MLLM。该框架首先在视觉编码器中实现FiCoCo-V，一种无需训练的方法，利用新型集成度量评估冗余性，精准剔除冗余视觉标记。为缓解信息丢失，引入基于相关性的信息回流机制，使保留标记可选择性地从相关联的被删标记中回收信息，实现自保持压缩，避免核心内容稀释。其变体FiCoCo-L进一步结合任务感知文本先验，在大语言模型解码器内直接进行标记压缩。大量实验表明，FiCoCo系列有效加速多种MLLM，在保持93.6%性能的同时实现最高达14.7倍的浮点运算量（FLOPs）降低。方法在不同模型架构、规模与任务下均优于现有最先进无训练方法，展现出强有效性与通用性，且无需重新训练。
原文摘要 · Abstract (English)
The quadratic complexity of Multimodal Large Language Models (MLLMs) with respect to context length poses significant computational and memory challenges, hindering their real-world deployment. In the paper, we devise a ''filter-correlate-compress'' framework to accelerate the MLLM by systematically optimizing multimodal context length during prefilling. The framework first implements FiCoCo-V, a training-free method operating within the vision encoder. It employs a redundancy-based token discard mechanism that uses a novel integrated metric to accurately filter out redundant visual tokens. To mitigate information loss, the framework introduces a correlation-based information recycling mechanism that allows preserved tokens to selectively recycle information from correlated discarded tokens with a self-preserving compression, thereby preventing the dilution of their own core content. The framework's FiCoCo-L variant further leverages task-aware textual priors to perform token reduction directly within the LLM decoder. Extensive experiments demonstrate that the FiCoCo series effectively accelerates a range of MLLMs, achieves up to 14.7x FLOPs reduction with 93.6% performance retention. Our methods consistently outperform state-of-the-art training-free approaches, showcasing effectiveness and generalizability across model architectures, sizes, and tasks without requiring retraining. Code: https://github.com/kawhiiiileo/FiCoCo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。