arXiv:2602.12172cs.AIcs.CL2026-02中稿 · ICLR被引 2

模仿教学规律生成合成数据,让小模型学得更像大模型。

Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation

  • 按教学规律分三阶段:找短板、循序渐进组织知识、适配认知能力。
  • 小模型性能达大模型94.7%,参数量不足十分之一,推理题提升19.2%。
  • 适合想高效训练小型模型的开发者,尤其擅长复杂逻辑任务。

将大型语言模型(LLMs)的知识压缩到小型模型,是实现高效AI部署的关键技术。现有基于合成数据的蒸馏方法缺乏教学意识,将知识传递视为一次性数据生成与训练任务。本文提出一种受教育学启发的新型蒸馏框架——知识识别器、组织者与适配器(IOA),借鉴布鲁姆掌握学习原则和维果茨基最近发展区理论,构建动态蒸馏流程:先识别学生模型的知识短板,再通过渐进式课程组织知识传递,最后调整表示以匹配其认知能力。实验使用LLaMA-3.1/3.2和Qwen2.5作为学生模型,结果表明,相比基线方法,该框架在保留教师模型94.7%性能的同时,参数量不足十分之一;在复杂推理任务上,于MATH数据集提升19.2%,于HumanEval提升22.3%。

原文摘要 · Abstract (English)

Knowledge distillation from Large Language Models (LLMs) to smaller models has emerged as a critical technique for deploying efficient AI systems. However, current methods for distillation via synthetic data lack pedagogical awareness, treating knowledge transfer as a one-off data synthesis and training task rather than a systematic learning process. In this paper, we propose a novel pedagogically-inspired framework for LLM knowledge distillation that draws from fundamental educational principles. Our approach introduces a three-stage pipeline -- Knowledge Identifier, Organizer, and Adapter (IOA) -- that systematically identifies knowledge deficiencies in student models, organizes knowledge delivery through progressive curricula, and adapts representations to match the cognitive capacity of student models. We integrate Bloom's Mastery Learning Principles and Vygotsky's Zone of Proximal Development to create a dynamic distillation process where student models approach teacher model's performance on prerequisite knowledge before advancing, and new knowledge is introduced with controlled, gradual difficulty increments. Extensive experiments using LLaMA-3.1/3.2 and Qwen2.5 as student models demonstrate that IOA achieves significant improvements over baseline distillation methods, with student models retaining 94.7% of teacher performance on DollyEval while using less than 1/10th of the parameters. Our framework particularly excels in complex reasoning tasks, showing 19.2% improvement on MATH and 22.3% on HumanEval compared with state-of-the-art baselines.

知识蒸馏教学启发小模型推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。