让大模型在稀疏奖励下自主突破难题，提升推理能力。
EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning
- 自生成并验证思维链轨迹，动态控制探索空间
- 使大模型能稳定学习从未解决的难题，提升推理准确率
- 无需外部监督，适配多种大模型与强化学习方法
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型（LLMs）推理能力的有前景范式。然而，在困难问题上推理准确率低时，奖励变得稀疏，导致学习效率下降并引发探索瓶颈。现有方法或依赖教师模型进行知识蒸馏，或过滤难例，限制了可扩展性或探索性改进。本文提出EvoCoT，一种基于两阶段思维链（CoT）优化的自进化课程学习框架。该框架通过自动生成并验证CoT轨迹来约束探索空间，随后逐步缩短CoT步数以受控方式拓展空间。该方法使大模型能在稀疏奖励下稳定学习初始无法解决的难题。我们在Qwen、DeepSeek和Llama等多个大模型家族上应用EvoCoT，实验表明其能使模型解决以往未解问题，提升推理能力且无需外部CoT监督，并兼容多种强化学习微调方法。源代码已公开，以支持后续研究。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable reward (RLVR) has become a promising paradigm for post-training large language models (LLMs) to improve their reasoning capability. However, when the rollout accuracy is low on hard problems, the reward becomes sparse, limiting learning efficiency and causing exploration bottlenecks. Existing approaches either rely on teacher models for distillation or filter out difficult problems, which limits scalability or restricts reasoning improvement through exploration. We propose EvoCoT, a self-evolving curriculum learning framework based on two-stage chain-of-thought (CoT) reasoning optimization. EvoCoT constrains the exploration space by self-generating and verifying CoT trajectories, then gradually shortens CoT steps to expand the space in a controlled way. The framework enables LLMs to stably learn from initially unsolved hard problems under sparse rewards. We apply EvoCoT to multiple LLM families, including Qwen, DeepSeek, and Llama. Experiments show that EvoCoT enables LLMs to solve previously unsolved problems, improves reasoning capability without external CoT supervision, and is compatible with various RL fine-tuning methods. We release the source code to support future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。