arXiv:2601.07155cs.LGcs.AI2026-01ACL被引 36

提出自适应目标重构方法,解决学生模型训练不稳问题。

Stable On-Policy Distillation through Adaptive Target Reformulation

  • 在logit空间构建中间目标,桥接师生模型分布差距
  • 通过可调参数抑制低置信度词的有害梯度,提升训练稳定性
  • 适合需要稳定生成且保持多样性的语言模型压缩场景

知识蒸馏(KD)广泛用于将大模型知识迁移到小模型;但传统监督式蒸馏常因训练与推理分布不匹配而效果下降。虽有在线策略蒸馏尝试直接从学生模型输出学习,但因初学者学生与专家教师间分布差距过大,易导致训练不稳定,表现为前向KL目标下的病态梯度或反向KL中的多样性崩溃。为此,本文提出Veto——一种目标层面的重构方法,在logit空间构建几何桥梁。不同于混合样本的方法,Veto生成一个中间目标分布,促进师生对齐。引入可调参数beta,使Veto既能作为自适应梯度抑制器,通过压制低置信度标记的有害梯度来稳定优化,又能作为决断旋钮,在奖励驱动性能与输出多样性之间实现平衡。在多种推理与生成任务上的大量实验表明,Veto在所有测试场景中均优于监督微调及现有在线策略基线。

原文摘要 · Abstract (English)

Knowledge distillation (KD) is a widely adopted technique for transferring knowledge from large language models to smaller student models; however, conventional supervised KD often suffers from a distribution mismatch between training and inference. While on-policy KD approaches attempt to mitigate this issue by learning directly from student-generated outputs, they frequently encounter training instabilities because the distributional gap between the novice student and the expert teacher is often too wide to bridge directly. These challenges manifest as pathological gradients in forward KL objectives or diversity collapse in reverse KL regimes. To address these limitations, we propose Veto, an objective-level reformulation that constructs a geometric bridge in the logit space. Unlike prior methods that mix data samples, Veto creates an intermediate target distribution that promotes alignment between the teacher and the student. By introducing a tunable parameter beta, Veto serves as an Adaptive Gradient Veto that stabilizes optimization by suppressing harmful gradients on low-confidence tokens, while simultaneously acting as a Decisiveness Knob to balance reward-driven performance with output diversity. Extensive experiments across various reasoning and generation tasks demonstrate that Veto consistently outperforms supervised fine-tuning and existing on-policy baselines.

知识蒸馏模型压缩训练稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。