用文本信息指导视觉标记恢复,压缩90%标记仍保持高精度。
Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information

- 根据问题与视觉标记的相似性,动态恢复关键视觉信息。
- 在仅保留10%原始视觉标记时,性能接近原模型。
- 无需训练,适合高效部署于多模态大模型中。
随着大规模语言建模技术的发展,结合视觉编码器与大语言模型的多模态模型在各类视觉任务中表现出色。当前主流方法将视觉编码器提取的视觉特征映射至大语言模型作为输入,与文本共同完成下游任务,因此视觉标记数量直接影响模型训练与推理速度。尽管已有大量关于视觉变换器标记剪枝的研究,但仅依赖视觉信息进行剪枝或压缩可能导致重要信息丢失。相比之下,以问题形式呈现的文本输入可能包含有助于回答问题的宝贵信息,为模型提供额外知识。为避免纯视觉标记剪枝导致的过度简化与过量剪枝,我们提出一种无需训练的文本信息引导动态视觉标记恢复机制:利用问题文本与视觉标记间的相似性,恢复带有重要文本关联的视觉标记,同时合并其余不重要标记。实验表明,该方法在将视觉标记压缩至平均原始量的10%时,性能与原始方法相当。源代码将在论文被接受后公开。
原文摘要 · Abstract (English)
With the advancement of large-scale language modeling techniques, large multimodal models combining visual encoders with large language models have demonstrated exceptional performance in various visual tasks. Most of the current large-scale multimodal models achieve this by mapping visual features obtained from the visual encoder into a large language model and using them as inputs alongside text for downstream tasks. Therefore, the number of visual tokens directly affects the training and inference speed of the model. There has been significant work on token pruning for visual transformers, but for large multimodal models, only relying on visual information for token pruning or compression may lead to significant loss of important information. On the other hand, the textual input in the form of a question may contain valuable information that can aid in answering the question, providing additional knowledge to the model. To address the potential oversimplification and excessive pruning that can occur with most purely visual token pruning methods, we propose a text information-guided dynamic visual token recovery mechanism that does not require training. This mechanism leverages the similarity between the question text and visual tokens to recover visually meaningful tokens with important text information while merging other less important tokens. Experimental results demonstrate that our proposed method achieves comparable performance to the original approach while compressing the visual tokens to an average of 10% of the original quantity. Our source code will be made publicly available following acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。