arXiv:2507.07685cs.CVcs.AI2025-07中稿 · CVPR

让视觉语言模型更重视推理过程,提升多模态思考的准确性。

Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought

  • 通过强化推理过程与视觉信息的关联,改进解码策略。
  • 在多个基准上显著提升多模态推理准确率,优于标准CoT方法。
  • 适用于希望提升模型可解释性和推理可信度的研究者。

大型视觉语言模型(LVLMs)通过融合预训练视觉编码器与大语言模型(LLMs)展现出强大能力。类似单模态大模型,多模态思维链(CoT)提示已被引入,基于视觉和文本输入生成中间推理过程。尽管假设CoT能增强模型的感知对齐与准确性,我们的实验发现:现有LVLMs常忽略生成的推理内容。为此,我们将多模态CoT推理重新建模为以推理条件似然为核心的KL约束奖励最大化问题。提出一种即插即用的推理时解码策略——理由增强解码(RED),通过乘法融合图像条件与推理条件的下一个词分布,协调视觉与推理信息。大量实验表明,RED在多个基准和多种LVLM上均显著优于标准CoT及其他解码方法。本工作提供了一种实用高效的方法,提升多模态思维链推理的忠实性与准确性,推动更可靠的基于推理的多模态系统发展。代码已开源:https://github.com/yshinya6/red/。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) have demonstrated remarkable capabilities by integrating pre-trained vision encoders with large language models (LLMs). Similar to single-modal LLMs, chain-of-thought (CoT) prompting has been adapted for LVLMs to enhance multi-modal reasoning by generating intermediate rationales based on visual and textual inputs. While CoT is assumed to improve grounding and accuracy in LVLMs, our experiments reveal a key challenge: existing LVLMs often ignore the contents of generated rationales in CoT reasoning. To address this, we re-formulate multi-modal CoT reasoning as a KL-constrained reward maximization focused on rationale-conditional log-likelihood. As the optimal solution, we propose rationale-enhanced decoding (RED), a novel plug-and-play inference-time decoding strategy. RED harmonizes visual and rationale information by multiplying distinct image-conditional and rationale-conditional next token distributions. Extensive experiments show that RED consistently and significantly improves reasoning over standard CoT and other decoding methods across multiple benchmarks and LVLMs. Our work offers a practical and effective approach to improve both the faithfulness and accuracy of CoT reasoning in LVLMs, paving the way for more reliable rationale-grounded multi-modal systems. Code is available at https://github.com/yshinya6/red/.

多模态思维链推理增强视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。