让教师模型在关键处干预学生生成，提升长文本生成质量
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
- 通过对比师生模型概率差异，动态决定教师何时介入
- 在五个数据集上显著优于传统知识蒸馏，长序列生成效果提升明显
- 适合需要高质量长文本生成的场景，如对话、摘要任务
尽管大型语言模型（LLMs）取得成功，但仍面临推理成本高和内存需求大的挑战。为应对这些问题，知识蒸馏（KD）成为主流模型压缩方法，其中使用学生生成输出（SGOs）作为训练数据尤为突出，可减少训练与推理间的不匹配。然而，SGOs常产生噪声大、有偏的序列，尤其在长序列中易导致教师模型误导。为此，我们提出SWITCH（Studying WIth TeaCHer for Knowledge Distillation），一种新方法，在学生生成序列时战略性地引入教师模型。SWITCH通过识别师生模型在词元概率上的差异，选择性地让教师干预，尤其针对更易受误导的长序列。在三个模型家族和五个指令遵循数据集上的实验表明，SWITCH超越传统KD方法，尤其在长序列生成方面表现优异。
原文摘要 · Abstract (English)
Despite the success of Large Language Models (LLMs), they still face challenges related to high inference costs and memory requirements. To address these issues, Knowledge Distillation (KD) has emerged as a popular method for model compression, with student-generated outputs (SGOs) as training data being particularly notable for reducing the mismatch between training and inference. However, SGOs often produce noisy and biased sequences, which can lead to misguidance from the teacher model, especially in long sequences. To mitigate these challenges, we propose SWITCH (Studying WIth TeaCHer for Knowledge Distillation), a novel approach that strategically incorporates the teacher model during the student's sequence generation. SWITCH identifies discrepancies between the token probabilities of the teacher and student models, allowing the teacher to intervene selectively, particularly in long sequences that are more prone to teacher misguidance. Extensive experimental results across three model families and five instruction-following datasets show that SWITCH surpasses traditional KD methods, particularly excelling in the generation of long sequential data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。