通过计算图文相似度，剔除无关图像标记，提升多模态模型复杂推理能力。
Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation
- 基于图文嵌入相似度，识别并忽略无关图像标记。
- 在复杂推理任务上显著提升模型表现，如VQA、OK-VQA等数据集。
- 方法轻量高效，适用于主流多模态大模型，提升可解释性。
多模态大语言模型发展迅速，但其内部机制仍不透明，尤其在链式思维推理等复杂任务中如同黑箱。研究发现，在LLaVA1.5等模型中，与文本语义相关的图像标记更易在语言模型解码层产生信息流汇聚，并获得更高注意力得分；而无关图像标记则无此现象且注意力极低。为此，本文提出新方法Simignore，通过计算图文嵌入相似度，主动忽略对文本无关的图像标记，以更高效利用图像信息。大量实验表明，该方法显著提升了模型在复杂推理任务中的性能，涵盖VQA、OK-VQA等数据集。代码已开源：https://github.com/FanshuoZeng/Simignore。
原文摘要 · Abstract (English)
Multimodal large language models have experienced rapid growth, and numerous different models have emerged. The interpretability of LVLMs remains an under-explored area. Especially when faced with more complex tasks such as chain-of-thought reasoning, its internal mechanisms still resemble a black box that is difficult to decipher. By studying the interaction and information flow between images and text, we noticed that in models such as LLaVA1.5, image tokens that are semantically related to text are more likely to have information flow convergence in the LLM decoding layer, and these image tokens receive higher attention scores. However, those image tokens that are less relevant to the text do not have information flow convergence, and they only get very small attention scores. To efficiently utilize the image information, we propose a new image token reduction method, Simignore, which aims to improve the complex reasoning ability of LVLMs by computing the similarity between image and text embeddings and ignoring image tokens that are irrelevant and unimportant to the text. Through extensive experiments, we demonstrate the effectiveness of our method for complex reasoning tasks. The paper's source code can be accessed from \url{https://github.com/FanshuoZeng/Simignore}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。