用奖励引导的门控机制,让视觉推理更准更稳。
Advantage-Guided Gate: Reshaping Open-Ended Reasoning for Vision-Based Spatial Intelligence

- 通过价值评估动态筛选优质推理路径和步骤。
- 在16万条推理树数据上训练,显著提升准确率。
- 适合需要稳定视觉空间推理的研究者使用。
多模态大语言模型在复杂空间场景理解与推理任务中展现出巨大潜力,但其开放式推理过程易产生决策错误与误差累积,导致答案质量不稳定。为此,我们提出一种优势引导的门控框架,动态干预并修正推理过程中的偏差。具体而言,将逐步推理建模为有限时域决策过程,并在推理树上引入蒙特卡洛价值评估,提供中间监督信号。该框架包含步骤优势门与轨迹优势门,分别动态选择高价值推理步骤和高质量完整推理轨迹。训练阶段,利用多分支采样生成的推理树进行监督学习,并采用共享参数初始化与任务特定头部相结合的方式,实现跨任务鲁棒性与多样性。推理阶段,模型贪心选择高价值前缀步骤,并根据问题类型选取最优推理头,显著提升最终答案准确性。此外,我们构建了包含16万条样本的Reasoning-Tree-160k数据集,并在此基础上进行两阶段训练。大量实验表明,该优势引导门控框架能有效提升基准多模态大模型在基于视觉的空间理解与推理任务中的表现。代码已开源:https://github.com/LingLin-ll/Advantage-Guided-Gate。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have demonstrated significant potential in complex spatial scene understanding and reasoning tasks. However, their open-ended reasoning process is prone to decision errors and error accumulation, leading to instability in answer quality. To address this, we propose an advantage-guided gating framework that dynamically intervenes in and corrects deviations during the reasoning process. Specifically, we model step-by-step reasoning as a finite-horizon decision process and introduce Monte Carlo value evaluation on the reasoning tree to provide intermediate supervision signals. The framework includes Step-Advantage Gate and Trajectory-Advantage Gate, which dynamically select high-value reasoning steps and high-quality complete reasoning trajectories, respectively. During training, we perform supervised learning for the gates using reasoning trees generated via multi-branch sampling, and combine shared-parameter initialization with task-specific heads to achieve cross-task robustness and diversity. During inference, the model greedily selects high-value prefix reasoning steps while choosing the optimal reasoning head based on the problem type, thereby significantly improving the accuracy of the final answer. Furthermore, we constructed the Reasoning-Tree-160k dataset and performed two-stage learning on it. Extensive experiments demonstrate that this advantage-guided gating framework effectively enhances the performance of benchmark MLLMs in visual-based spatial understanding and reasoning tasks. The code is open to the public for research: https://github.com/LingLin-ll/Advantage-Guided-Gate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。