发现强化学习蒸馏中输出格式崩溃的临界点,可精准控制模型性能提升
The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs
- 提出可解析的阈值公式,预测蒸馏时输出格式安全边界
- 在亚马逊时尚数据集上实现1.7B模型与8B基线同等表现,参数量仅为五分之一
- 适用于结构化输出任务,尤其适合追求高效且格式正确的场景
在基于奖励外推系数λ > 1的在线策略蒸馏(OPD)中,学生模型虽能在域内超越教师模型,但当λ超过临界阈值λ*时,其生成的结构化输出将违反格式约束。通过单位置伯努利简化模型,推导出闭式解的剪裁安全阈值λ*(p,b,c),由教师模态概率、预热质量与重要性采样剪裁强度决定。当λ > λ*时,外推固定点移出剪裁安全区,训练从保持格式转为格式坍缩。该规则扩展至校准的K元列表式JSON任务,其中单一绑定等价类主导输出契约,且监督微调保留解析空间。在Amazon Fashion数据集上,三次预注册测试——细网格悬崖区间、预算扩展测试、小剪裁交叉预测——均落在锁定预测窗口内,小剪裁值与闭式预测一致,且低于网格分辨率。在λ*以下运行时,ListOPD使1.7B Qwen3学生模型达到与8B-SFT基线相同的域内表现,参数仅为其五分之一。性能增益主要源于格式合规性:解析输出的NDCG@1随λ变化平稳,而解析有效性在预测边界处急剧下降。悬崖诊断与评分标准无关,但性能对比使用Gemini评分,受其评估者暴露影响。
原文摘要 · Abstract (English)
On-policy distillation (OPD) is widely used for LLM post-training. When pushed with a reward-extrapolation coefficient lambda > 1, the student can lift past the teacher in domain, but past a threshold lambda* the same step violates the output contract on structured-output tasks. In a single-position Bernoulli reduction, we derive a closed-form base-relative clip-safety threshold lambda*(p,b,c) determined by three measurable quantities: the teacher modal probability, the warm-start mass, and the importance-sampling clip strength. Above lambda*, the extrapolated fixed point exits the clip-safe region, changing training from format-preserving to format-collapsing. We extend the rule to calibrated K-ary listwise JSON tasks where a single binding equivalence class dominates the output contract and SFT retains parse headroom. On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval, a budget-extension test, and a small-clip cross-prediction--fall within their locked prediction windows, with the small-clip value matching the closed-form prediction below grid resolution. Operating just below lambda*, ListOPD brings a 1.7B Qwen3 student to in-domain parity with an 8B-SFT baseline at one-fifth the parameters. The gain is driven primarily by format adherence: NDCG@1 on parsed outputs remains flat across lambda, while parse validity sharply changes at the predicted boundary. The cliff diagnostic is rubric-independent, whereas the parity claim uses a Gemini-graded rubric and inherits that evaluator's exposure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。