让大模型从自身错误中学习，无需专家指导就能持续提升推理能力。
Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error
- 用自身过往错误提示模型改进，避免陷入重复试错的僵局。
- 在6个数学推理数据集上，相比标准方法平均提升5.02（Pass@1）和9.96（Pass@k）。
- 不依赖外部专家，适合希望自主优化推理能力的研究者与开发者。
基于可验证奖励的强化学习（RLVR）显著提升了语言模型（LMs）的推理能力。然而，现有方法依赖模型自身的在线响应进行训练，受限于初始能力，容易陷入探索停滞——即无法解决更多问题，也无法从训练数据中继续学习。部分方法尝试通过使用离线策略解题缓解此问题，但需依赖外部专家指导，存在可用性与可扩展性瓶颈。本文提出LTE（Learning to Reason from Trial and Error），通过提示模型其先前自创的错误，实现无外部指导的自我改进。实验表明，LTE在六个数学推理基准上对Qwen3-8B-Base的表现优于标准组相对策略优化（GRPO），Pass@1平均提升5.02，Pass@k平均提升9.96，甚至超越依赖外部指导的方法。进一步分析证实，LTE有效缓解了探索停滞，同时增强了训练中的利用与探索能力。代码已开源：https://github.com/JamyDon/LTE。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has significantly boosted the reasoning capability of language models (LMs). However, existing RLVR approaches train LMs based on their own on-policy responses and are constrained by the initial capability of LMs, thus prone to exploration stagnation, in which LMs fail to solve more training problems and cannot further learn from the training data. Some approaches try to address this by leveraging off-policy solutions to training problems, but rely on external expert guidance that is limited in availability and scalability. In this work, we propose LTE (Learning to reason from Trial and Error), an approach that hints LMs with their previously self-made mistakes, not requiring any external expert guidance. Experiments validate the effectiveness of LTE, which outperforms the normal group relative policy optimization (GRPO) by 5.02 in Pass@1 and 9.96 in Pass@k on average across six mathematical reasoning benchmarks for Qwen3-8B-Base and even performs better than methods that require external guidance. Further analysis confirms that LTE successfully mitigates exploration stagnation and enhances both exploitation and exploration during training. Our code is available at https://github.com/JamyDon/LTE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。