提升大模型蒸馏的输出多样性，解决传统方法过自信问题。
Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
- 新方法移除非目标类梯度干扰，避免学生模型过度自信。
- 在多个数据集和模型上显著优于现有蒸馏目标，提升精度与多样性平衡。
- 适合关注模型泛化与输出多样性的研究人员使用。
反向Kullback-Leibler（RKL）散度近年来成为大语言模型蒸馏的首选目标，尤其在词汇量大且师生模型能力差异显著时表现更优，其通过聚焦主导模式而非强制密集对齐来优化学习。然而，RKL存在结构性缺陷，导致学生模型预测过于自信。本文首次将RKL梯度分解为目标与非目标成分，发现非目标梯度会持续推动目标词元向上，即使学生已匹配教师，从而削弱输出多样性；同时，对非目标类别的监督较弱，造成尾部类别对齐不佳。为此，我们提出多样性感知的RKL（DRKL），消除该梯度影响并强化非目标监督，同时保留RKL的优化优势。大量实验表明，DRKL在多个数据集与模型家族中持续优于FKL、RKL及其他先进蒸馏目标，实现更高性能与更优的保真度-多样性权衡。
原文摘要 · Abstract (English)
Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL), particularly in regimes with large vocabularies and significant teacher-student capacity mismatch, where RKL focuses learning on dominant modes rather than enforcing dense alignment. However, RKL introduces a structural limitation that drives the student toward overconfident predictions. We first provide an analysis of RKL by decomposing its gradients into target and non-target components, and show that non-target gradients consistently push the target logit upward even when the student already matches the teacher, thereby reducing output diversity. In addition, RKL provides weak supervision over non-target classes, leading to poor tail alignment. To address these issues, we propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL. Extensive experiments across datasets and model families demonstrate that DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity-diversity trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。