让AI推理过程与答案保持一致,提升多模态模型的逻辑可靠性。
Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models
- 引入一致性验证机制,通过打乱选项重测答案来检测推理与结论是否匹配。
- 在多模态视频和数学推理任务上分别提升2.2%和1.5%准确率。
- 特别适合需要严谨逻辑链的多模态问答场景,如教育评测、智能客服。
近期大型语言模型的发展表明,基于可验证奖励的强化学习(RLVR)可通过直接优化正确性显著提升推理能力,而非仅依赖监督模仿。该范式已拓展至复杂视频与图像理解任务的多模态大模型。然而,尽管结果导向的强化学习提升了答案准确率,却可能导致推理链与最终答案脱节,出现不一致现象。我们在多选视觉问答任务上的实验显示,标准GRPO方法在MMVU数据集上推理步骤与答案的一致性仅为79.7%,表明答案与推理之间存在频繁矛盾。为此,我们提出答案一致性强化学习(ACRE),对GRPO算法进行改进,增加辅助一致性检查。模型生成思维链与初始答案后,重新打乱选项并使用相同推理链预测第二个答案。设计一致性验证奖励:仅当原始答案与重测答案均正确且一致时给予高奖励,否则给予较低奖励。该机制惩罚推理与答案的错配,抑制模型对选项顺序等虚假模式的依赖。我们在挑战性视频推理与多模态数学推理基准上评估ACRE,相比GRPO基线,平均分别取得2.2%和1.5%的性能提升。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have demonstrated that reinforcement learning with verifiable rewards (RLVR) can significantly enhance reasoning abilities by directly optimizing correctness, rather than relying solely on supervised imitation. This paradigm has been extended to multimodal LLMs for complex video and image understanding tasks. However, while outcome-driven RL improves answer accuracy, it can inadvertently decouple the reasoning chain from the final answer, leading to situations where models produce inconsistency between the reasoning trace and final answer. In our experiments on multiple-choice visual question-answering tasks, the standard GRPO method yields only 79.7\% consistency on MMVU between the reasoning steps and the chosen answers, indicating frequent mismatches between answers and reasoning. To this end, we propose Answer-Consistent Reinforcement Learning (ACRE) that modifies the GRPO algorithm with an auxiliary consistency check. After the model generates a chain of thought and an initial answer for a given question, we shuffle the answer options and prompt the model again with the same reasoning trace to predict a second answer. We design a consistency-verification reward that grants a high reward only if both the original and the post-shuffle answers agree and are correct; otherwise, a lower reward is assigned accordingly. This mechanism penalizes reasoning-answer misalignment and discourages the model from relying on spurious patterns, such as option ordering biases. We evaluate ACRE on challenging Video Reasoning benchmarks and multimodal math reasoning benchmarks, achieving an average 2.2\% and 1.5\% improvement for Video Reasoning and Math Reasoning tasks over the GRPO baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。