通过分阶段模拟教师训练过程，让小模型在特定领域超越大模型。
Following the Teacher's Footsteps: Scheduled Checkpoint Distillation for Domain-Specific LLMs
- 分阶段模仿教师训练路径，缩小学生与教师的性能差距。
- 在多语言问答、命名实体识别等任务上，学生模型表现优于教师。
- 适合需要高效部署大模型能力的领域应用开发者使用。
大语言模型在特定领域任务中部署困难，因其规模庞大。虽然将微调后的教师模型蒸馏到更小的学生模型是可行方案，但师生间的容量差异常导致性能不佳。本文提出新理论：若学生在自身优势子领域（SFS）的表现超过其在教师优势子领域（TFS）的不足，则可超越教师。基于此，提出分阶段检查点蒸馏（SCD），通过模仿教师在领域任务上的监督微调收敛过程来减少TFS缺陷，并引入样本级自适应加权机制保留学生在SFS中的优势。在多种领域任务——包括多语言问答、命名实体识别和文本分类——上的实验表明，该方法持续优于现有蒸馏方法，使学生模型达到甚至超越其微调教师的性能。
原文摘要 · Abstract (English)
Large language models (LLMs) are challenging to deploy for domain-specific tasks due to their massive scale. While distilling a fine-tuned LLM into a smaller student model is a promising alternative, the capacity gap between teacher and student often leads to suboptimal performance. This raises a key question: when and how can a student model match or even surpass its teacher on domain-specific tasks? In this work, we propose a novel theoretical insight: a student can outperform its teacher if its advantage on a Student-Favored Subdomain (SFS) outweighs its deficit on the Teacher-Favored Subdomain (TFS). Guided by this insight, we propose Scheduled Checkpoint Distillation (SCD), which reduces the TFS deficit by emulating the teacher's convergence process during supervised fine-tuning (SFT) on the domain task, and a sample-wise Adaptive Weighting (AW) mechanism to preserve student strengths on SFS. Experiments across diverse domain tasks--including QA, NER, and text classification in multiple languages--show that our method consistently outperforms existing distillation approaches, allowing the student model to match or even exceed the performance of its fine-tuned teacher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。