根据学习进度动态调整训练滚展长度，显著提速且不损失效果
Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation
- 按学习进展判断是否扩展滚展长度，避免盲目延长
- 相比原方法训练时间减少49.1%至71.2%，平均性能最优
- 适合追求高效训练的模型蒸馏研究者和工程师
在线策略蒸馏（OPD）通过学生生成轨迹提供密集教师监督，但其在线滚展过程计算开销大，尤其当少数长响应延迟批次完成。现有加速方法通常使用固定预算或绝对教师-学生一致阈值控制滚展长度，难以反映不同模型与训练阶段的学习进展。本文提出自适应快速OPD（Adaptive FastOPD），仅在学习接近当前边界区域趋于停滞且当前滚展已充分使用时才扩展滚展。前者基于进入每个滚展时四个教师-学生信号的相对变化判断，使扩展响应于阶段性进展而非预设步数或原始一致信号的绝对阈值；后者防止少数长响应引发滚展成本上升。在两个教师-学生对上，Adaptive FastOPD 实现最高平均性能，同时相较 OPD 15K 减少 49.1%–71.2% 训练时间，并在多种超参数设置下保持鲁棒。
原文摘要 · Abstract (English)
On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher--student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher--student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher--student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1--71.2\% relative to OPD 15K, and remains robust across a range of hyperparameter settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。