提出分层视觉令牌压缩方法，显著提升多模态大模型效率
LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models
- 在视觉编码器中间层进行分层压缩，通过空间到通道变换合并相邻令牌
- 相比现有方法，训练效率提升20%以上，推理吞吐量提高15%以上
- 采用无参数残差捷径保留关键视觉信息，适合资源受限场景部署
现有的多模态大语言模型（MLLMs）视觉令牌压缩方法主要作为编码器后的模块运行，限制了其效率提升潜力。为此，我们提出LaCo（分层视觉令牌压缩）框架，实现视觉编码器中间层的有效令牌压缩。LaCo引入两个核心组件：1）分层像素重排机制，通过空间到通道变换系统性地合并相邻令牌；2）非参数化残差学习架构，压缩过程中保留关键视觉信息。大量实验表明，与现有方法相比，LaCo在视觉编码器中间层压缩时表现更优。相较于外部压缩，本方法训练效率提升超过20%，推理吞吐量提升超过15%，同时保持强性能。
原文摘要 · Abstract (English)
Existing visual token compression methods for Multimodal Large Language Models (MLLMs) predominantly operate as post-encoder modules, limiting their potential for efficiency gains. To address this limitation, we propose LaCo (Layer-wise Visual Token Compression), a novel framework that enables effective token compression within the intermediate layers of the vision encoder. LaCo introduces two core components: 1) a layer-wise pixel-shuffle mechanism that systematically merges adjacent tokens through space-to-channel transformations, and 2) a residual learning architecture with non-parametric shortcuts that preserves critical visual information during compression. Extensive experiments indicate that our LaCo outperforms all existing methods when compressing tokens in the intermediate layers of the vision encoder, demonstrating superior effectiveness. In addition, compared to external compression, our method improves training efficiency beyond 20% and inference throughput over 15% while maintaining strong performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。