用强化学习训练教师模型，提升推理模型的蒸馏效率。
Reinforcement Learning Teachers of Test Time Scaling
- 教师模型通过已知问题与答案生成解释，以连接线索方式指导学生。
- 70亿参数教师在竞赛和研究生级任务上超越更大模型的蒸馏效果。
- 无需微调即可跨任务复用，显著提升强化学习推理框架的效率。
使用强化学习（RL）训练推理型语言模型（LMs）时，通常依赖模型在初始化阶段具备一定探索和解题能力。此外，推理型语言模型的核心用途是作为教师，用于蒸馏新学生模型或冷启动未来的强化学习迭代，而非直接部署。基于此，我们提出一种新框架，通过训练一类新的强化学习教师（RLTs）来规避强化学习中的探索难题。这些教师模型接收问题和对应解法，任务是为学生生成详尽的“连接线索”式解释。训练过程中，通过将每段解释输入学生模型并测试其对解法的理解程度，获得密集奖励信号。实验表明，70亿参数的RLT原始输出，在竞赛及研究生级任务上的最终表现优于现有基于海量大模型推理轨迹收集与后处理的蒸馏与冷启动方法。此外，当应用于更大规模学生模型或零样本迁移至分布外任务时，该方法仍保持高效性，显著提升了强化学习推理框架的效率与可重用性。代码已开源：https://github.com/SakanaAI/RLT
原文摘要 · Abstract (English)
Training reasoning language models (LMs) with reinforcement learning (RL) for one-hot correctness inherently relies on the LM being able to explore and solve its task with some chance at initialization. Furthermore, a key use case of reasoning LMs is to act as teachers for distilling new students and cold-starting future RL iterations rather than being deployed themselves. From these considerations, we introduce a new framework that avoids RL's exploration challenge by training a new class of Reinforcement-Learned Teachers (RLTs) focused on yielding the most effective downstream distillation. RLTs are prompted with both the question and solution to each problem, and tasked to simply "connect-the-dots" with detailed explanations tailored for their students. We train RLTs with dense rewards obtained by feeding each explanation to the student and testing its understanding of the problem's solution. In practice, the raw outputs of a 7B RLT provide higher final performance on competition and graduate-level tasks than existing distillation and cold-starting pipelines that collect and postprocess the reasoning traces of orders of magnitude larger LMs. Furthermore, RLTs maintain their effectiveness when training larger students and when applied zero-shot to out-of-distribution tasks, unlocking new levels of efficiency and re-usability for the RL reasoning framework. Code available at: https://github.com/SakanaAI/RLT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。