arXiv:2509.21016cs.LGcs.CL2025-09被引 22

用强化学习让大模型学会全新解题策略,并测试其迁移能力。

RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?

  • 设计新基准DELTA-Code,通过模板生成问题来分离推理技能。
  • 训练中出现突变式'领悟'现象:长期无进展后突然准确率飙升至接近完美。
  • 验证了策略可跨任务迁移,但面对根本性创新仍存局限,适合研究模型泛化。

大语言模型能否在预训练或微调之外真正习得或泛化出全新的推理策略,仍是未解之谜。为此,我们提出DELTA-Code——一种用于评估算法编码中可学性与可迁移性的受控基准。该基准采用模板化问题生成器,构建出需新策略而非工具调用或记忆的合成问题家族。实验揭示显著的‘领悟’相变:在长时间几乎无奖励后,强化学习训练模型突然实现近乎完美的准确率。为实现对原有无法解决的问题家族的学习,我们探索了分阶段预热、密集奖励、经验回放、课程学习及验证闭环等关键训练要素。在可学性之外,通过探索性、组合性与变革性维度评估迁移能力,结果表明在家族内和重组技能上表现良好,但在变革性任务中仍有明显短板。DELTA为探究强化学习驱动推理的边界提供了清晰测试平台,有助于理解模型如何突破现有先验获取新算法能力。

原文摘要 · Abstract (English)

It remains an open question whether LLMs can acquire or generalize genuinely new reasoning strategies, beyond the sharpened skills encoded in their parameters during pre-training or post-training. To attempt to answer this debate, we introduce DELTA-Code -- Distributional Evaluation of Learnability and Transferrability in Algorithmic Coding -- a controlled benchmark of synthetic coding problem families designed to probe two fundamental aspects: learnability -- can LLMs, through reinforcement learning (RL), solve problem families where pretrained models exhibit failure with large enough attempts (pass@K=0)? -- and transferrability -- if learnability happens, can such skills transfer systematically to out-of-distribution (OOD) test sets? Unlike prior public coding datasets, DELTA isolates reasoning skills through templated problem generators and introduces fully OOD problem families that demand novel strategies rather than tool invocation or memorized patterns. Our experiments reveal a striking grokking phase transition: after an extended period with near-zero reward, RL-trained models abruptly climb to near-perfect accuracy. To enable learnability on previously unsolvable problem families, we explore key training ingredients such as staged warm-up with dense rewards, experience replay, curriculum training, and verification-in-the-loop. Beyond learnability, we use DELTA to evaluate transferability or generalization along exploratory, compositional, and transformative axes, as well as cross-family transfer. Results show solid gains within families and for recomposed skills, but persistent weaknesses in transformative cases. DELTA thus offers a clean testbed for probing the limits of RL-driven reasoning and for understanding how models can move beyond existing priors to acquire new algorithmic skills.

强化学习算法推理模型泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。