让大模型自己生成学习路径,突破推理瓶颈
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
- 用自洽教学框架让模型生成适合自己的练习题
- 在0/128成功率的数学难题上实现有效学习
- 题目设计质量比答案正确更重要,适合研究模型自进化
大型推理模型在初始成功率极低的数据集上训练时,因缺乏有效信号而停滞。我们探究一个根本问题:预训练的大语言模型能否利用隐含知识生成自身无法解决的问题的自动化学习课程?提出SOAR框架——一种非对称自对弈机制,通过元强化学习挖掘教学信号。教师模型为学生模型设计合成问题,其奖励基于学生在部分难题上的进步,使课程建立在真实学习进展上,而非内在代理奖励。在最难数学基准(0/128成功率)上的研究揭示三个核心发现:第一,可通过精炼预训练模型生成有用问题的潜在能力,实现稀疏二值奖励下的双层元强化学习;第二,基于真实进展的奖励优于以往使用可学习性代理奖励的方法,稳定且避免多样性崩溃;第三,问题结构与良构性对学习进展的影响大于解的正确性。结果表明,生成有效学习阶梯无需预先具备解决难题的能力,为突破推理瓶颈提供了原则性路径,无需额外标注数据。
原文摘要 · Abstract (English)
RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? We explore this with SOAR: An asymmetric self-play framework that uses meta-RL to surface these pedagogical signals. A teacher model proposes synthetic problems for a student model, and is rewarded with its improvement on a subset of hard problems, thus grounding the curriculum in real student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of math benchmarks (0/128 success) reveals three core findings. First, it is possible to realize bilevel meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful problems. Second, grounded rewards outperform intrinsic learnability rewards used in prior LLM self-play, reliably avoiding typical instability and diversity collapse modes. Third, the structure and well-posedness of questions are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。