arXiv:2602.11549cs.LGcs.AI2026-02中稿 · ICLR被引 2

让大模型自己生成推理路径,无需人工标注和外部验证。

Native Reasoning Models: Training Language Models to Reason on Unverifiable Data

  • 用自生成推理轨迹替代人工标注,训练时无需外部验证器。
  • 在多个复杂推理任务上超越现有无验证器方法,显著提升准确率。
  • 适合构建通用、可扩展的推理系统,尤其擅长处理不可验证任务。

当前大型推理模型的训练范式依赖高质量人工标注的推理数据和外部验证器,导致成本高昂、易引入认知偏见,且仅适用于数学与编程等可验证领域。为突破此限制,我们提出NRT(原生推理训练)框架,通过仅使用标准问答对,让模型自主生成推理过程,无需专家示范或外部验证。NRT将推理视为潜在变量,以统一目标函数建模推理为优化问题,内在奖励那些提高模型产出正确答案概率的推理路径。该视角揭示了先前方法的内在失败模式(如策略坍塌),并系统设计出更鲁棒的奖励聚合函数,形成自我强化反馈回路,使模型学会解决自身不确定性。在Llama与Mistral系列模型上的实证表明,NRT在无验证器方法中达到最先进水平,显著优于标准SFT基线及以往无验证器强化学习方法,在复杂推理任务中表现尤为突出,且对策略坍塌具有高鲁棒性,为构建更强大、广泛适用的推理系统提供了通用且可扩展的路径。

原文摘要 · Abstract (English)

The prevailing paradigm for training large reasoning models--combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR)--is fundamentally constrained by its reliance on high-quality, human-annotated reasoning data and external verifiers. This dependency incurs significant data-collection costs, risks embedding human cognitive biases, and confines the reinforcement learning stage to objectively assessable domains like mathematics and coding, leaving a wide range of unverifiable tasks beyond its scope. To overcome these limitations, we introduce NRT (Native Reasoning Training), a novel framework that cultivates complex reasoning by having the model generate its own reasoning traces using only standard question-answer pairs, thereby obviating the need for expert-written demonstrations. NRT reframes the training problem by treating the reasoning process as a latent variable. It employs a unified training objective that models reasoning as an optimization problem, intrinsically rewarding paths that increase the model's likelihood of producing the ground-truth answer. This unified perspective allows us to analyze intrinsic failure modes of prior methods, such as policy collapse, and systematically design more robust reward aggregation functions, creating a self-reinforcing feedback loop where the model learns to think in ways that resolve its own uncertainty. Empirical evaluation on Llama and Mistral model families demonstrates that NRT achieves state-of-the-art performance among verifier-free methods, significantly outperforming standard SFT baselines and prior verifier-free RL methods. Our approach yields particularly strong performance gains in complex reasoning domains and exhibits high robustness to policy collapse, offering a general, scalable path toward building more powerful and broadly applicable reasoning systems.

推理模型自生成无验证大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。