提出新方法提升大模型知识蒸馏效率，大幅降低内存与存储开销。
Rethinking Selective Knowledge Distillation
- 按位置、类别、样本三维度分离选择性蒸馏，分析不同策略效果。
- 引入学生熵引导的位置选择，实现准确率不降的前提下减少70%训练时间。
- 扩展至三轴选择，使离线教师缓存可行，存储节省80%且不影响性能。
为提升大语言模型中知识蒸馏（KD）的效果，现有研究趋向于用选择性蒸馏替代密集型教师监督，即仅在部分词元位置、词汇类别或训练样本上进行监督。然而，何种重要性信号、选择策略及其相互作用最有效仍不明确。本文重新审视自回归大语言模型中的蒸馏时机与方式，将选择性蒸馏解耦为位置、类别和样本三个维度，并系统比较不同重要性信号与选择策略。基于此分析，我们发现未充分探索的机会，并提出学生熵引导的位置选择（SE-KD）。在多个基准测试中，SE-KD在保持精度的同时，优于密集蒸馏的下游任务契合度与内存效率。将该方法拓展至类别与样本维度（SE-KD 3X），带来互补的效率提升，使离线教师缓存成为可能：实际应用中，训练耗时减少70%，峰值内存降低18%，存储使用量减少80%，且性能无损。
原文摘要 · Abstract (English)
Growing efforts to improve knowledge distillation (KD) in large language models (LLMs) replace dense teacher supervision with selective distillation, which uses a subset of token positions, vocabulary classes, or training samples for supervision. However, it remains unclear which importance signals, selection policies, and their interplay are most effective. In this work, we revisit where and how to distill in autoregressive LLMs. We disentangle selective KD along the position, class, and sample axes and systematically compare importance signals and selection policies. Then, guided by this analysis, we identify underexplored opportunities and introduce student-entropy-guided position selection (SE-KD). Across a suite of benchmarks, SE-KD often improves accuracy, downstream task adherence, and memory efficiency over dense distillation. Extending this approach across the class and sample axes (SE-KD 3X) yields complementary efficiency gains that make offline teacher caching feasible. In practice, this reduces wall time by 70% and peak memory by 18%, while cutting storage usage by 80% over prior methods without sacrificing performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。