通过推理链条件优化,显著减少多模态大模型的幻觉。
Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

- 将推理链作为答案生成的条件,分步优化推理与答案一致性。
- 在多个基准上幻觉率下降30%以上,答案可靠性显著提升。
- 适合关注多模态推理可信度的研究者和开发者。
多模态大推理模型引入推理范式,在复杂视觉语言任务中表现强劲,但仍存在严重幻觉问题。现有基于训练的方法通常采用响应级直接偏好优化(DPO),将思维链(CoT)与最终答案视为整体输出联合优化。我们发现该方法性能接近仅优化答案的方案,表明其主要学习答案层面偏好,而未能充分挖掘推理链层面的监督信息。为此,我们明确提出一种面向推理链的偏好项,并推导出推理条件直接偏好优化(RC-DPO),将推理链建模为答案生成的条件,对比同一优选答案在不同推理链下的偏好差异,促进答案与推理链的一致性对齐。为进一步提升优化效果,我们设计了一种增强推理的偏好数据生成策略:利用蒙特卡洛树搜索发现视觉一致且逻辑连贯的推理链作为正样本,结合注意力引导的推理链标记剪枝构建负样本。在多种模型与基准上的大量实验表明,RC-DPO能有效缓解幻觉问题,提升多模态推理过程的可靠性。
原文摘要 · Abstract (English)
Multimodal Large Reasoning Models introduce the reasoning paradigm, demonstrating strong capabilities on complex vision-language tasks. However, they still suffer from severe hallucinations. Existing training-based methods typically mitigate hallucinations through response-level direct preference optimization (DPO), where the Chain-of-Thought (CoT) and the final answer are treated as a monolithic output and optimized jointly. We reveal that this formulation performs similarly to answer-only optimization, suggesting that it primarily learns answer-level preference, while leaving CoT-level supervision insufficiently exploited. To address this issue, we explicitly formulate a CoT-oriented preference term and derive Reasoning-Conditioned Direct Preference Optimization (RC-DPO), which models the CoT as a condition for answer generation and contrasts the preference for the same preferred answer under different CoT conditions, promoting answer-supportive reasoning chain alignment. To further improve optimization, we introduce a reasoning-enhanced preference data generation strategy that employs Monte Carlo Tree Search to discover visually grounded and logically consistent CoTs as positive samples, and attention-guided CoT token pruning to construct negative ones. Extensive experiments across various models and benchmarks show that RC-DPO effectively mitigates hallucinations and improves the reliability of the multimodal reasoning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。