让视觉语言模型更聪明地看图说话,自动发现关键细节并适时引用。
AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning
- 用上下文增强注意力图缓解图文粒度不平衡问题
- 通过信息觅食主动挑选最相关图像证据
- 动态触发插入时机,匹配模型关注点转变
交错模态思维链(I-MCoT)推动了视觉语言推理发展,如视觉问答(VQA)。该范式将输入图像中精选的视觉证据融入视觉语言模型(VLM)上下文,使其推理逻辑基于具体细节。然而现有方法在证据选择和插入触发两方面均存在不足:一是依赖不可靠的注意力信号,尤其在文本查询简短而图像信息丰富时;二是采用静态触发机制,无法捕捉VLM对视觉证据的动态需求。为此,我们提出新型I-MCoT框架——主动信息驱动多模态思维链(AIM-CoT),通过三项改进提升性能:(1) 上下文增强注意力图生成(CAG),通过文本上下文增强缓解粒度失衡;(2) 主动视觉探查(AVP),通过信息觅食过程主动选取最具信息量的证据;(3) 动态注意力转移触发(DAT),在VLM注意力从文本转向视觉时精准激活插入。在三个基准数据集和四种骨干网络上的实验表明,AIM-CoT表现持续领先。代码已公开于https://anonymous.4open.science/r/AIMCoT。
原文摘要 · Abstract (English)
Interleaved-Modal Chain-of-Thought (I-MCoT) advances vision-language reasoning, such as Visual Question Answering (VQA). This paradigm integrates specially selected visual evidence from the input image into the context of Vision-Language Models (VLMs), enabling them to ground their reasoning logic in these details. Accordingly, the efficacy of an I-MCoT framework relies on identifying what to see (evidence selection) and when to see it (triggering of insertions). However, existing methods fall short in both aspects. First, for selection, they rely on attention signals, which are unreliable -- particularly under severe granularity imbalance between the brief textual query and the informative image. Second, for triggering, they adopt static triggers, which fail to capture the VLMs' dynamic needs for visual evidence. To this end, we propose a novel I-MCoT framework, Active Information-driven Multi-modal Chain-of-Thought (AIM-CoT), which aims to improve both evidence selection and insertion triggering via: (1) Context-enhanced Attention-map Generation (CAG) to mitigate granularity imbalance via textual context enhancement; (2) Active Visual Probing (AVP) to proactively select the most informative evidence via an information foraging process; and (3) Dynamic Attention-shift Trigger (DAT) to precisely activate insertions when VLM's attention shifts from text to visual context. Experiments across three benchmarks and four backbones demonstrate AIM-CoT's consistent superiority. Our code is available at https://anonymous.4open.science/r/AIMCoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。