arXiv:2605.11260cs.LGcs.AI2026-05

让小模型学得更好:按难易顺序+逐步换更强老师

Curriculum Learning-Guided Progressive Distillation in Large Language Models

论文配图:Curriculum Learning-Guided Progressive Distillation in Large Language Models
图 1 · 摘自论文原文
  • 按题目难易排序训练数据,同时逐步使用更强的教师模型
  • 在推理任务上,比传统方法提升显著,强教师不再无效
  • 适合想高效蒸馏大模型推理能力的研究者和工程师

知识蒸馏是将大语言模型(LLM)能力迁移到更小、更高效的学生模型的关键技术。现有方法常忽略两个重要因素:训练数据的学习顺序与教师-学生模型之间的能力差异。这一忽视导致蒸馏效果受限,表现为强教师反而无法生成更好的学生模型。本文提出课程学习引导的渐进式蒸馏(CLPD),通过将数据难度与教师强度对齐,显式构建从易到难的数据课程,同时隐式构建由弱到强教师的监督信号调度。该框架模块化设计,可无缝集成至标准蒸馏算法且开销极低。在多个推理基准上的实验证明,CLPD在不同设置下均优于标准蒸馏、仅数据排序或仅教师调度的方法。结果强调了在蒸馏推理能力时,需同时考虑数据顺序与教师容量。

原文摘要 · Abstract (English)

Knowledge distillation is a key technique for transferring the capabilities of large language models (LLMs) into smaller, more efficient student models. Existing distillation approaches often overlook two critical factors: the learning order of training data and the capacity mismatch between teacher and student models. This oversight limits distillation performance, as manifested by the counter-intuitive phenomenon where stronger teachers fail to produce better students. In this work, we propose Curriculum Learning-Guided Progressive Distillation (CLPD), a unified framework that explicitly accounts for both factors by aligning data difficulty with teacher strength. CLPD constructs an explicit curriculum by organizing training examples from easy to hard, while simultaneously applying an implicit curriculum over supervision signals by progressively scheduling teachers of increasing capacity. Our framework is modular and can be integrated into standard distillation algorithms with minimal overhead. Empirical results on the reasoning benchmarks demonstrate that CLPD consistently outperforms standard distillation, data ordering alone, and teacher scheduling alone across multiple settings. These findings highlight the importance of jointly considering data ordering and teacher capacity when distilling reasoning abilities into small language models.

知识蒸馏大模型压缩课程学习推理能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。