通过融合视觉令牌压缩序列，提升多模态模型推理效率。
Learning Compact Vision Tokens for Efficient Large Multimodal Models
- 用空间融合方法减少视觉令牌数量，缩短序列长度。
- 仅用25%的视觉令牌，在8个基准上达到或超越基线性能。
- 适合需要高效推理的多模态应用开发者使用。
大型多模态模型（LMMs）因大语言模型（LLMs）和视觉令牌序列的二次复杂度而面临显著计算挑战。本文探索视觉令牌中的空间冗余，通过缩短视觉令牌序列来加速推理。提出空间令牌融合（STF）方法，将空间相邻的令牌融合为一个紧凑表示。同时，冻结权重的视觉编码器难以适应广泛的下游视觉-语言任务，为此引入多块令牌融合（MBTF）模块，为压缩后的令牌序列补充多粒度特征。结合STF与MBTF，在减少令牌数量的同时保留关键信息，实现推理效率提升而不损失多模态推理能力。实验表明，基于LLaVA-1.5的方法仅使用25%的视觉令牌，就在8个主流视觉-语言基准上达到或优于基线表现。源代码与训练权重已公开于https://github.com/visresearch/LLaVA-STF。
原文摘要 · Abstract (English)
Large multimodal models (LMMs) suffer significant computational challenges due to the high cost of Large Language Models (LLMs) and the quadratic complexity of processing long vision token sequences. In this paper, we explore the spatial redundancy among vision tokens and shorten the length of vision token sequences for inference acceleration. Specifically, we propose a Spatial Token Fusion (STF) method to learn compact vision tokens for short vision token sequence, where spatial-adjacent tokens are fused into one. Meanwhile, weight-frozen vision encoder can not well adapt to the demand of extensive downstream vision-language tasks. To this end, we further introduce a Multi-Block Token Fusion (MBTF) module to supplement multi-granularity features for the reduced token sequence. Overall, we combine STF and MBTF module to balance token reduction and information preservation, thereby improving inference efficiency without sacrificing multimodal reasoning capabilities. Experimental results demonstrate that our method based on LLaVA-1.5 achieves comparable or even superior performance to the baseline on 8 popular vision-language benchmarks with only $25\%$ vision tokens of baseline. The source code and trained weights are available at https://github.com/visresearch/LLaVA-STF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。