用噪声推理前缀训练模型自我纠错，无需强监督也能提升推理能力。
DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes
- 从弱模型的错误推理中学习，通过修复噪声前缀来优化
- 在数学和交互决策任务中超越主流强化学习基线
- 适合想提升大模型自主纠错能力的研究者
强化学习已成为推动大语言模型推理能力的核心范式，但现有方法仍依赖更强的教师模型或精心构建的困难数据集，限制了可扩展的能力提升。本文提出DenoiseRL，一种以故障恢复为导向的强化学习框架，将外部监督替换为对弱模型失败结果的优化。该方法直接从噪声推理前缀中学习，将其转化为改进机会，并能精细控制噪声强度，使训练更具可扩展性和有效性。这带来了更丰富多样的学习信号，提升了探索效率，利用了不完美的模型行为。实验表明，DenoiseRL在多个具有挑战性的数学推理任务和交互决策任务上持续优于强大的在线策略强化学习基线，且随着训练难度增加，自修正行为更强，展示了提升大模型智能体推理能力的有效且可扩展路径。
原文摘要 · Abstract (English)
Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from noisy reasoning prefixes by converting them into opportunities for improvement, while exercising fine-grained control over the noise intensity, making training more scalable and effective. This yields a richer and more diverse learning signal, improving exploration efficiency by leveraging imperfect model behavior. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical reasoning tasks and interactive decision-making tasks, while promoting stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable pathway for improving agentic reasoning capabilities of large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。