用反馈迭代优化小模型数学推理能力,提升性能同时降低资源消耗。
Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
- 根据小模型表现分类题目,生成难易适配的训练数据
- 多轮迭代增强数据集,使10亿参数小模型达到顶尖数学推理水平
- 适合部署在手机、边缘设备等资源受限场景
大语言模型(LLMs)展现出卓越的推理能力,常在各类任务中达到领先性能。然而,其数十亿参数带来的巨大计算与内存开销,限制了在资源受限环境中的部署。知识蒸馏是一种有前景的解决方案,通过将LLMs的推理能力迁移到小型语言模型(SLMs,≤1B参数),实现在低资源设备上的广泛部署。现有方法主要关注高质量推理链的生成,却忽视了数据量与质量的协同作用。为此,我们提出反馈驱动蒸馏(FDD)框架,以提升SLMs的数学推理能力。在初始化阶段,利用LLMs为数学问题生成对应推理链,构建蒸馏数据集;根据SLM表现将问题分为易题和难题:对易题生成更复杂的变体,对难题合成同复杂度的新题。此外,采用多轮蒸馏范式,迭代丰富蒸馏数据集,逐步提升SLMs的数学推理能力。实验结果表明,该方法使SLMs在数学推理任务上达到最先进水平。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate exceptional reasoning capabilities, often achieving state-of-the-art performance in various tasks. However, their substantial computational and memory demands, due to billions of parameters, hinder deployment in resource-constrained environments. A promising solution is knowledge distillation, where LLMs transfer reasoning capabilities to Small Language Models (SLMs, $\le$ 1B parameters), enabling wider deployment on low-resource devices. Existing methods primarily focus on generating high-quality reasoning rationales for distillation datasets but often neglect the critical role of data quantity and quality. To address these challenges, we propose a Feedback-Driven Distillation (FDD) framework to enhance SLMs' mathematical reasoning capabilities. In the initialization stage, a distillation dataset is constructed by prompting LLMs to pair mathematical problems with corresponding reasoning rationales. We classify problems into easy and hard categories based on SLM performance. For easy problems, LLMs generate more complex variations, while for hard problems, new questions of similar complexity are synthesized. In addition, we propose a multi-round distillation paradigm to iteratively enrich the distillation datasets, thereby progressively improving the mathematical reasoning abilities of SLMs. Experimental results demonstrate that our method can make SLMs achieve SOTA mathematical reasoning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。