让大模型自己检查并修正数学推理错误，无需外部评分。
Self-rewarding correction for mathematical reasoning
- 用自生成数据训练模型，实现推理与自我评估一体化。
- 在Llama-3和Qwen-2.5上表现超越原生自校正能力。
- 适合需要低成本部署的自主推理系统研究者。
我们研究自奖励推理大型语言模型（LLMs），这类模型能在推理过程中同时生成逐步推理并评估自身输出的正确性，无需外部反馈。这种集成方法使单个模型能独立引导其推理过程，有利于模型部署的计算效率。我们重点关注自修正任务，即模型自主检测回答中的错误、修改输出，并决定何时终止迭代优化循环。为此，我们提出一种仅使用自生成数据的两阶段算法框架。第一阶段通过序列拒绝采样构建包含自奖励与自修正机制的长链式推理轨迹，利用这些精炼数据微调模型以学习相关模式。第二阶段通过基于规则信号的强化学习进一步提升模型对响应准确性的判断力和输出优化能力。在Llama-3和Qwen-2.5上的实验表明，该方法超越了内在自修正能力，性能接近依赖外部奖励模型的系统。
原文摘要 · Abstract (English)
We study self-rewarding reasoning large language models (LLMs), which can simultaneously generate step-by-step reasoning and evaluate the correctness of their outputs during the inference time-without external feedback. This integrated approach allows a single model to independently guide its reasoning process, offering computational advantages for model deployment. We particularly focus on the representative task of self-correction, where models autonomously detect errors in their responses, revise outputs, and decide when to terminate iterative refinement loops. To enable this, we propose a two-staged algorithmic framework for constructing self-rewarding reasoning models using only self-generated data. In the first stage, we employ sequential rejection sampling to synthesize long chain-of-thought trajectories that incorporate both self-rewarding and self-correction mechanisms. Fine-tuning models on these curated data allows them to learn the patterns of self-rewarding and self-correction. In the second stage, we further enhance the models' ability to assess response accuracy and refine outputs through reinforcement learning with rule-based signals. Experiments with Llama-3 and Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction capabilities and achieves performance comparable to systems that rely on external reward models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。