通过蒸馏缩小电信领域问答模型，优化师生模型微调策略提升性能。
Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom
- 先对教师或学生模型进行领域微调，再进行知识蒸馏。
- 教师模型微调+相同词表可显著提升蒸馏后模型表现。
- 同时微调师生模型效果最佳，适合追求高精度的工业应用。
知识蒸馏（KD）是缩小大语言模型（LLM）规模的有效方法：用参数更少的学生模型模仿参数更多的教师模型在特定任务上的表现。针对电信领域的问答（QA）任务，当前尚不清楚应仅对教师、仅对学生，还是两者都进行领域适配。本文系统研究了仅教师微调、仅学生微调以及两者均微调三种策略在知识蒸馏前的效果。实验还考察了词表一致与不一致，以及标准蒸馏与双空间蒸馏（DSKD）算法的影响。采用14种评估指标（包括N-gram、嵌入和基于LLM的度量）进行多维度评测。结果表明，当两模型使用相同词表时，教师模型微调能显著提升蒸馏后模型性能；整体而言，同时对教师和学生模型进行微调在所有指标上表现更优，尽管统计显著性依赖于教师模型的词表设置。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) is one of the approaches to reduce the size of Large Language Models (LLMs). A LLM with smaller number of model parameters (student) is trained to mimic the performance of a LLM of a larger size (teacher model) on a specific task. For domain-specific tasks, it is not clear if teacher or student model, or both, must be considered for domain adaptation. In this work, we study this problem from perspective of telecom domain Question-Answering (QA) task. We systematically experiment with Supervised Fine-tuning (SFT) of teacher only, SFT of student only and SFT of both prior to KD. We design experiments to study the impact of vocabulary (same and different) and KD algorithms (vanilla KD and Dual Space KD, DSKD) on the distilled model. Multi-faceted evaluation of the distillation using 14 different metrics (N-gram, embedding and LLM-based metrics) is considered. Experimental results show that SFT of teacher improves performance of distilled model when both models have same vocabulary, irrespective of algorithm and metrics. Overall, SFT of both teacher and student results in better performance across all metrics, although the statistical significance of the same depends on the vocabulary of the teacher models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。