无需外部验证器的强化学习框架，让模型自动优化推理过程。
NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning
- 用监督微调数据替代验证器，实现无验证器的激励训练
- 在多类文本任务上性能超越6710亿参数大模型7.7%
- 支持反向激励训练等新范式，适用范围更广
近期如DeepSeek R1-Zero的方法展示了激励训练的有效性，该方法仅根据语言模型输出的最终答案部分计算奖励，从而鼓励生成中间推理步骤。然而，这些方法依赖外部验证器，在数学和编程等领域的适用性受限。尽管可使用奖励模型充当验证器，但需高质量标注数据且训练成本高。本文提出NOVER（No-Verifier Reinforcement Learning），一种仅需标准监督微调数据、无需外部验证器的通用强化学习框架。NOVER可在广泛文本到文本任务中实现激励训练，其性能优于同规模从大型推理模型（如DeepSeek R1 671B）蒸馏出的模型7.7个百分点。此外，NOVER的灵活性为优化大模型提供了新可能，例如逆向激励训练。
原文摘要 · Abstract (English)
Recent advances such as DeepSeek R1-Zero highlight the effectiveness of incentive training, a reinforcement learning paradigm that computes rewards solely based on the final answer part of a language model's output, thereby encouraging the generation of intermediate reasoning steps. However, these methods fundamentally rely on external verifiers, which limits their applicability to domains like mathematics and coding where such verifiers are readily available. Although reward models can serve as verifiers, they require high-quality annotated data and are costly to train. In this work, we propose NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning framework that requires only standard supervised fine-tuning data with no need for an external verifier. NOVER enables incentive training across a wide range of text-to-text tasks and outperforms the model of the same size distilled from large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the flexibility of NOVER enables new possibilities for optimizing large language models, such as inverse incentive training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。