arXiv:2607.27770cs.LG2026-07

通过构建互补教师群体,让小模型超越最强单个教师。

Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold

论文配图:Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
图 1 · 摘自论文原文
  • 先扩展后压缩:用多轮训练覆盖更多解题模式
  • 学生在数学、代码生成等任务上提升2%-8.3%
  • 适合追求高效推理的小模型应用

单一强化学习训练只能挖掘部分有效解题路径,我们提出先扩展后压缩的框架。扩张阶段使用残差组相对策略优化(RGRPO)从同一初始化出发,逐步引导后续教师覆盖未被涵盖的样本;压缩阶段采用可靠性门控教师联合在线策略蒸馏(TU-OPD),仅允许可靠教师贡献,其损失按例级质量加权。引入共识残差分解,保留优教师对其他教师的额外偏好,避免专长被压制。在数学推理、代码生成和指令遵循任务上,所得Qwen3-1.7B学生模型均优于最强单教师,相对提升分别为2.0%、8.3%、6.9%,且保持单模型推理。结果表明,更强的学生来自互补教师集合的构造与压缩,而非选择单一更优教师。

原文摘要 · Abstract (English)

A single reinforcement-learning run can produce a strong reasoner yet an incomplete teacher: it often amplifies only a subset of the valid solution modes. We argue that reinforcement learning (RL)-trained policies should therefore be viewed as local probes of a multi-basin reasoning solution manifold, rather than as globally reliable supervisors. Based on this view, we propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation. In the expansion stage, Residual Group Relative Policy Optimization (RGRPO) trains a sequence of teachers from a common initialization and redirects each later round toward examples not yet covered by the accumulated teacher union. In the compression stage, reliability-gated Teacher-Union On-policy Distillation (TU-OPD) lets the student learn from its own response prefixes. For each example, only reliable teachers contribute, and their sampled-token OPD losses are weighted by their per-example quality. We further introduce Consensus-Residual Decomposition, which preserves a winner teacher's excess token preferences over its reliable peers, preventing specialist behavior from being suppressed during teacher aggregation. Experiments on mathematical reasoning, code generation, and instruction following show that the resulting Qwen3-1.7B student consistently outperforms the strongest individual teacher across all three domains, yielding relative improvements of 2.0%, 8.3%, and 6.9%, respectively, while retaining single-model inference. These results establish a simple but powerful principle: stronger students can be obtained not by selecting a single better teacher, but by deliberately constructing and compressing a complementary teacher union.

强化学习知识蒸馏推理模型教师聚合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。