FOLDER通过删减70%视觉令牌，加速多模态大模型推理且不损失性能。
FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance
- 设计可即插即用的令牌压缩模块，保留关键视觉信息
- 在多个模型上实现推理速度提升，最高减少70%视觉令牌
- 既适合作为推理加速器，也可提升训练效率
多模态大语言模型（MLLMs）在跨模态任务中表现出色，但视觉骨干网络提取的长序列视觉令牌给实时应用部署带来挑战。为此，我们提出FOLDER，一种简单有效的即插即用模块，用于缩短视觉令牌序列，降低训练与推理阶段的计算和内存开销。通过对不同压缩策略的信息损失分析，FOLDER在去除冗余的同时保留关键信息。我们将FOLDER集成到多个MLLM的视觉骨干中，显著加速推理过程。此外，评估表明其在训练加速或性能增强方面均表现优异：相比原模型，性能相当甚至更优，同时最多可移除70%的视觉令牌，大幅降低复杂度。
原文摘要 · Abstract (English)
Recently, Multi-modal Large Language Models (MLLMs) have shown remarkable effectiveness for multi-modal tasks due to their abilities to generate and understand cross-modal data. However, processing long sequences of visual tokens extracted from visual backbones poses a challenge for deployment in real-time applications. To address this issue, we introduce FOLDER, a simple yet effective plug-and-play module designed to reduce the length of the visual token sequence, mitigating both computational and memory demands during training and inference. Through a comprehensive analysis of the token reduction process, we analyze the information loss introduced by different reduction strategies and develop FOLDER to preserve key information while removing visual redundancy. We showcase the effectiveness of FOLDER by integrating it into the visual backbone of several MLLMs, significantly accelerating the inference phase. Furthermore, we evaluate its utility as a training accelerator or even performance booster for MLLMs. In both contexts, FOLDER achieves comparable or even better performance than the original models, while dramatically reducing complexity by removing up to 70% of visual tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。