用教师模型找出关键词，帮小模型更好理解任务。
Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights
- 教师模型识别输入中影响决策的关键词，作为理由传给学生模型。
- 在四个数据集上表现优于标准微调和现有蒸馏方法。
- 68%情况下关键词包含正确答案，适合需要可解释性的场景。
提升小型语言模型在实际应用中的性能是当前研究的重要挑战。由于大型语言模型的使用成本高且困难，研究人员正寻求高效部署特定任务的小型模型。本文提出一种简单而有效的知识蒸馏方法，利用约30亿参数的教师模型识别其决策过程中最具影响力的输入词元。这些词元通过显著性图等方法，根据其对输出的归因分数提取，并作为推理依据提供给学生模型，以实现教师模型知识的提炼。该方法在四个不同数据集上的测试表明，其性能优于标准微调及当前最先进的知识蒸馏模型。此外，我们分析了模型成功的原因，发现68%的情况下，从教师模型中提取的关键词属于真实答案，尤其在标签为答案一部分的任务（如多项选择题）中表现明显。此结果验证了该方法在提升小模型准确性和可解释性方面的有效性。
原文摘要 · Abstract (English)
Enhancing small language models for real-life application deployment is a significant challenge facing the research community. Due to the difficulties and costs of using large language models, researchers are seeking ways to effectively deploy task-specific small models. In this work, we introduce a simple yet effective knowledge distillation method to improve the performance of small language models. Our approach utilizes a teacher model with approximately 3 billion parameters to identify the most influential tokens in its decision-making process. These tokens are extracted from the input based on their attribution scores relative to the output, using methods like saliency maps. These important tokens are then provided as rationales to a student model, aiming to distill the knowledge of the teacher model. This method has proven to be effective, as demonstrated by testing it on four diverse datasets, where it shows improvement over both standard fine-tuning methods and state-of-the-art knowledge distillation models. Furthermore, we explore explanations of the success of the model by analyzing the important tokens extracted from the teacher model. Our findings reveal that in 68\% of cases, specifically in datasets where labels are part of the answer, such as multiple-choice questions, the extracted tokens are part of the ground truth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。