用少量精选数据实现高效推理模型蒸馏,兼顾性能与泛化能力。
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
- 基于强化学习启发的教师选择与多样化推理路径设计
- 仅用800个样本即达当前最优推理效果
- 适合追求高效推理训练的研究者与开发者
大语言模型在算法编程和数学问题求解等任务中展现出卓越推理能力。尽管已有方法通过扩大语料库和多阶段训练提升推理表现,但推理缩放规律尚未明确,计算成本仍高。为此,我们提出一种数据高效的蒸馏框架(DED),优化推理蒸馏的帕累托前沿。受强化学习中在线策略学习与多样化采样启发,核心思路为:(1)发现基准分数不能单独决定有效教师模型,通过对比主流推理模型,提出教师模型优选方法;(2)大规模蒸馏常损害跨领域性能,而精心筛选的小型语料库可在域内与域外能力间实现平衡;(3)多样化推理轨迹促进学生模型发展稳健推理技能。我们在数学推理(AIME 2024/2025、MATH-500)与代码生成(LiveCodeBench)上验证该方法,仅使用0.8k个精心筛选样本即达到领先水平,无需大规模扩展。系统分析表明,DED通过考虑超越表面难度、词元长度和教师能力的因素,优于现有方法。本工作为高级推理提供了一条实用高效的路径,同时保持模型通用性。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving. Recent methods have improved reasoning through expanded corpus and multistage training combining reinforcement learning and supervised fine-tuning. Although some methods suggest that small but targeted dataset can incentivize reasoning via only distillation, a reasoning scaling laws is still taking shape, increasing computational costs. To address this, we propose a data-efficient distillation framework (DED) that optimizes the Pareto frontier of reasoning distillation. Inspired by the on-policy learning and diverse roll-out strategies of reinforcement learning, the key idea of our approach is threefold: (1) We identify that benchmark scores alone do not determine an effective teacher model. Through comprehensive comparisons of leading reasoning LLMs, we develop a method to select an optimal teacher model. (2) While scaling distillation can enhance reasoning, it often degrades out-of-domain performance. A carefully curated, smaller corpus achieves a balanced trade-off between in-domain and out-of-domain capabilities. (3) Diverse reasoning trajectories encourage the student model to develop robust reasoning skills. We validate our method through evaluations on mathematical reasoning (AIME 2024/2025, MATH-500) and code generation (LiveCodeBench), achieving state-of-the-art results with only 0.8k carefully curated examples, bypassing the need for extensive scaling. Our systematic analysis demonstrates that DED outperforms existing methods by considering factors beyond superficial hardness, token length, or teacher model capability. This work offers a practical and efficient pathway to advanced reasoning while preserving general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。