解决大模型训练中因生成过长重复文本导致的崩溃问题
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
- 用参考分布约束+混合回溯蒸馏，抑制学生模型生成过长重复序列
- 在多个数学推理数据集上平均提升7.2%性能，防止训练崩溃
- 适合追求稳定高效强化学习蒸馏的模型开发者
在线策略蒸馏（OPD）在学生模型自身生成的数据分布下训练，同时利用更强教师模型的监督信号。我们发现OPD存在一种失败模式：随着训练进行，在线策略采样会出现突发性长度膨胀，导致截断轨迹主导训练数据。这种截断坍塌伴随重复饱和现象，引发偏差梯度信号，造成严重训练不稳定和验证性能骤降。我们归因于学生生成数据与蒸馏目标之间的相互作用，该机制隐式偏好长且重复的生成序列。为此，我们提出StableOPD框架，结合基于参考的分布差异约束与回溯混合蒸馏，有效缓解重复引起的长度膨胀，并进一步稳定训练过程。在多个数学推理数据集上，该方法可防止截断坍塌，稳定训练动态，并实现平均7.2%的性能提升。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。