arXiv:2512.06258cs.CV2025-12被引 7

LVLMs常答对但路走歪,新方法通过纠错路径优化提升推理稳定性。

Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs

  • 用两阶段后训练框架,引导模型选择逻辑一致的推理路径。
  • 平均推理准确率提升7.4%,且错误路径被记忆并反复修正。
  • 适合关注视觉语言模型可靠推理的开发者与研究者。

我们揭示了大型视觉-语言模型(LVLMs)中一个关键但未被充分探讨的问题:即使模型知道正确答案,也常通过错误的推理路径得出结果。根本原因并非知识缺失,而是庞大推理搜索空间中的路径选择偏差。尽管模型能采样出正确解题轨迹,却更倾向于不稳定或逻辑不一致的路径,导致结果不可靠。多个模型在大K值下的通过率(Pass@K)远高于单次采样通过率(Pass@1),有力证明此类失败主要源于误推理而非无知。为此,我们提出路径选择优化(PSO)框架,分两阶段改进现有LVLM的推理性能与稳定性。第一阶段采用基于模板和答案奖励的组相对策略优化(GRPO),培养结构化、逐步推理能力;第二阶段进行在线偏好优化,模型从GRPO生成数据中采样路径,自我评估并对齐至优选路径,同时将错误路径存入负向回放记忆(NRM)作为硬负样本,定期重访以避免重复错误,实现持续推理优化。大量实验表明,PSO有效剔除无效路径,显著提升推理准确率(平均提高7.4%),并产生更稳定一致的思维链。

原文摘要 · Abstract (English)

We reveal a critical yet underexplored flaw in Large Vision-Language Models (LVLMs): even when these models know the correct answer, they frequently arrive there through incorrect reasoning paths. The core issue is not a lack of knowledge, but a path selection bias within the vast reasoning search space. Although LVLMs are often capable of sampling correct solution trajectories, they disproportionately favor unstable or logically inconsistent ones, leading to erratic and unreliable outcomes. The substantial disparity between Pass@K (with large K) and Pass@1 across numerous models provides compelling evidence that such failures primarily stem from misreasoning rather than ignorance. To systematically investigate and address this issue, we propose PSO (Path-Select Optimization), a two-stage post-training framework designed to enhance both the reasoning performance and stability of existing LVLMs. In the first stage, we employ Group Relative Policy Optimization (GRPO) with template and answer-based rewards to cultivate structured, step-by-step reasoning. In the second stage, we conduct online preference optimization, where the model samples reasoning paths from GRPO-generated data, self-evaluates them, and aligns itself toward the preferred trajectories. Incorrect or suboptimal paths are concurrently stored in a Negative Replay Memory (NRM) as hard negatives, which are periodically revisited to prevent the model from repeating prior mistakes and to facilitate continual reasoning refinement. Extensive experiments show that PSO effectively prunes invalid reasoning paths, substantially enhances reasoning accuracy (with 7.4% improvements on average), and yields more stable and consistent chains of thought. Our code will be available at https://github.com/aiming-lab/PSO.

视觉语言模型推理优化路径选择后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。