用物体级压缩减少图像令牌，提升多模态大模型效率
AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
- 基于物体感知的自适应令牌合并策略
- 仅用10%令牌达到原模型96%性能
- 适合追求高效推理的多模态应用
多模态大语言模型在统一图文理解与推理方面表现出巨大价值，主要通过将图像转换为与架构一致的补丁级令牌序列实现。然而，补丁级令牌化导致图像令牌数量呈二次增长，给模型的理解与推理带来巨大的计算和内存负担。此外，传统的补丁扫描令牌化流程与人类视觉认知系统不匹配，进一步引发幻觉和计算冗余。为此，我们提出一种面向自适应令牌压缩的物体级令牌合并策略，揭示其与人类视觉系统的内在一致性。在多个综合性基准上的实验表明，我们的方法平均仅使用10%的令牌，便达到了原始模型约96%的性能。与其他相关工作相比，实验结果充分证明了该方法在压缩率与性能之间的优越平衡。代码将公开。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated substantial value in unified text-image understanding and reasoning, primarily by converting images into sequences of patch-level tokens that align with their architectural paradigm. However, patch-level tokenization leads to a quadratic growth in image tokens, burdening MLLMs' understanding and reasoning with enormous computation and memory. Additionally, the traditional patch-wise scanning tokenization workflow misaligns with the human vision cognition system, further leading to hallucination and computational redundancy. To address this issue, we propose an object-level token merging strategy for Adaptive Token compression, revealing the consistency with human vision system. The experiments are conducted on multiple comprehensive benchmarks, which show that our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance. More extensive experimental results in comparison with relevant works demonstrate the superiority of our method in balancing compression ratio and performance. Our code will be available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。