让小模型推理又快又准,还能按需控制思考长度
BARD: budget-aware reasoning distillation
- 用两阶段训练让小模型学会根据预算调整推理过程
- 80亿参数模型在多个难题上表现接近大模型,且推理长度可控
- 适合需要高效推理的部署场景,如移动端或实时系统
尽管长链式思维(CoT)蒸馏能有效将推理能力迁移到小型语言模型,但推理过程常冗余且计算预算不可控,导致资源浪费。为此,我们提出预算感知推理蒸馏(BARD),一种同时蒸馏推理能力并实现细粒度推理长度控制的新框架。BARD将思考预算作为用户指定的控制信号,使模型动态平衡推理性能与计算效率。该框架采用两阶段训练:第一阶段在教师生成的长链式思维数据上进行监督微调,将其压缩至不同预算水平,以建立模型对预算约束的理解;第二阶段利用强化学习,基于推理性能与预算契合度双重奖励信号进行优化。两阶段结合可避免策略退化,确保双目标协同优化。大量实验表明,该方法使80亿参数的学生模型在挑战性推理基准(AIME24、AIME25、GPQA)上表现优异,且能在广泛预算范围内实现精确自适应的推理长度控制。
原文摘要 · Abstract (English)
While long Chain-of-Thought (CoT) distillation effectively transfers reasoning capability to smaller language models, the reasoning process often remains redundant and computational budget uncontrollable, leading to inefficient resource usage. To address this limitation, we propose \textbf{Budget-Aware Reasoning Distillation (BARD)}, a novel framework that simultaneously distills reasoning capability and enables fine-grained control over the reasoning length. BARD uses the thinking budget as a user-specified control signal, allowing the model to dynamically balance reasoning performance and computational efficiency. To achieve this concept, BARD introduces a two-phase training regimen. The first phase, Supervised Fine-Tuning (SFT) on teacher-generated long CoT data compressed to various budget levels, bootstrapping the model's understanding of budget constraints. The second phase leverages Reinforcement Learning (RL) from a reward signal in consideration of reasoning performance and budget fidelity simultaneously. Incorporating the two-phase regimen is crucial to avoiding policy degradation and ensuring that both objectives are optimized jointly. Extensive experiments demonstrate that our method empowers an 8B student model to achieve strong performance on challenging reasoning benchmarks (\textit{AIME24, AIME25, GPQA}) while providing precise and adaptive control over its reasoning length across a wide range of budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。