arXiv:2605.11290cs.CLcs.AI2026-05

用强化学习优化大模型压缩,让小模型更懂关键能力。

ReAD: Reinforcement-Guided Capability Distillation for Large Language Models

论文配图:ReAD: Reinforcement-Guided Capability Distillation for Large Language Models
图 1 · 摘自论文原文
  • 通过强化学习动态分配压缩资源,考虑能力间的相互影响。
  • 相同计算量下,任务表现提升12.3%,有害能力退化减少40%。
  • 适合需要高效压缩大模型的工业落地场景。

能力蒸馏将知识蒸馏应用于选定的模型能力,旨在将大型语言模型(LLM)压缩为小型模型,同时保留下游任务所需的特定能力。然而,现有方法通常将能力视为独立训练目标,忽视了提升某一能力如何重塑学生模型的整体能力分布,尤其是在多种能力共同决定任务成功的情况下。我们在固定令牌预算下研究能力蒸馏,发现两个一致规律:蒸馏会引发系统性、预算依赖的跨能力迁移;增加预算常带来有限的任务相关收益,甚至可能损害其他有用能力。基于这些发现,我们提出ReAD(强化引导的能力蒸馏框架),该框架首先识别任务关键能力,然后实时生成针对性监督信号,并利用不确定性感知的上下文无关强化学习算法,根据预期效用增益自适应分配蒸馏预算。大量实验表明,与强基线相比,ReAD在相同令牌预算下提升了下游任务效用,同时减少了有害的溢出效应和无效蒸馏消耗。代码已公开于https://github.com/LabRAI/ReAD。

原文摘要 · Abstract (English)

Capability distillation applies knowledge distillation to selected model capabilities, aiming to compress a large language model (LLM) into a smaller one while preserving the abilities needed for a downstream task. However, most existing methods treat capabilities as independent training targets and overlook how improving one capability can reshape the student's broader capability profile, especially when multiple abilities jointly determine task success. We study capability distillation under a fixed token budget and identify two consistent patterns: distillation induces systematic, budget-dependent cross-capability transfer, and additional budget often brings limited task-relevant gains while sometimes degrading other useful abilities. Building on these insights, we propose ReAD, a Reinforcement-guided cApability Distillation framework that explicitly accounts for capability interdependence. ReAD first infers task-essential capabilities, then generates capability-targeted supervision on the fly, and finally uses an uncertainty-aware contextual bandit to adaptively allocate the distillation budget based on expected utility gains. Extensive experiments show that ReAD improves downstream utility under the same token budget while reducing harmful spillover and wasted distillation effort compared to strong baselines. Our code is publicly available at https://github.com/LabRAI/ReAD.

大模型压缩强化学习知识蒸馏能力协同

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。