用奖励机制优化小模型推理能力，提升数学与复杂任务表现
Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
- 基于教师模型多轮生成结果，用规则验证器分配奖励作为训练权重
- 在数学和复杂推理任务上，学生模型性能显著提升，超越传统蒸馏方法
- 适合需要高效推理的小模型部署场景，尤其关注逻辑与数学能力
将大语言模型（LLMs）的能力压缩并迁移到更高效的小语言模型（SLMs）中，依赖知识蒸馏（KD）技术。然而，传统蒸馏仅让小模型复制教师模型的分布内输出，限制了其泛化能力，尤其在推理任务中表现不足且计算成本高。本文提出AdvDistill，一种基于奖励的蒸馏框架：对每个提示生成多个教师响应，通过规则验证器分配奖励，利用这些呈正态分布的奖励作为训练加权。实验表明，该方法显著提升了学生模型在数学与复杂推理任务上的表现，证明奖励机制能有效增强数据蒸馏的效果。
原文摘要 · Abstract (English)
The push to compress and impart the proficiency of Large Language Models (LLMs) into more deployable and efficient Small Language Models (SLMs) has benefited from improvements in knowledge distillation (KD) techniques. These techniques allow a smaller student model to learn from a more capable and larger teacher model's responses. However, distillation often revolves around the student model merely copying the teacher's in-distribution responses, limiting its generalisability. This limitation is amplified on reasoning tasks and can be computationally expensive. In this study, we propose AdvDistill, a reward-guided dataset distillation framework. We utilise multiple generations (responses) from a teacher for each prompt and assign rewards based on rule-based verifiers. These varying and normally distributed rewards serve as weights when training student models. Our methods and their subsequent behavioural analysis demonstrate a significant improvement in student model performance for mathematical and complex reasoning tasks, showcasing the efficacy and benefits of incorporating a rewarding mechanism in dataset distillation processes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。