arXiv:2605.22675cs.CL2026-05被引 3

无需外部信号,精准提升大模型特定能力。

Self-Policy Distillation via Capability-Selective Subspace Projection

  • 从模型梯度中提取能力子空间,定向投影生成内容。
  • 在代码与数学推理任务中,性能比现有方法高13%。
  • 无需额外标注,通用性强,适合前沿大模型优化。

自蒸馏通过模型自身生成内容进行训练,但现有方法要么依赖外部信号(如正确性过滤、执行反馈)筛选输出,成本高且不适用于顶尖模型;要么跳过筛选,直接训练所有原始输出,存在领域局限且难以泛化。两者共同问题在于自生成内容混杂了任务相关能力与风格、格式错误等无关因素,稀释了目标能力的信号。本文提出自策略蒸馏(SPD),实现无需外部信号的可泛化、能力选择性提升。具体地,SPD从模型对正确性关键标记的梯度中提取低秩能力子空间,生成时将键值(KV)激活投影至该子空间,并使用标准下一个词预测损失进行微调。在代码生成、数学推理和多项选择题问答任务上的大量实验表明,SPD相比无外部信号的最先进自蒸馏方法性能提升最高达13%,相比预训练基线提升最高达16%。尤其在跨领域泛化设置下,表现优于基线15%。

原文摘要 · Abstract (English)

Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedback, and reward search), which are costly and unavailable for the best-performing frontier models, or skip curation entirely and train on all raw outputs, an approach that is often domain-specific and hard to generalize. Both also share a deeper weakness that self-generated outputs entangle task-relevant capability with others, such as stylistic patterns, formatting artifacts, and model-specific errors, diluting the signal for the specific capability one aims to improve. In this paper, we propose Self-Policy Distillation (SPD), which achieves generalizable, capability selective without any external signal. Specifically, SPD extracts a low-rank capability subspace from the model's own gradients on correctness-defining tokens, projects key-value (KV) activations into this subspace during self-generation, and fine-tunes on the resulting raw outputs with standard next-token prediction loss. Through extensive experiments across code generation, mathematical reasoning, and multiple-choice QA, we show that SPD achieves up to 13% improvement over state-of-the-art self-distillation methods without external signals and up to 16% improvement over pre-trained baselines. Notably, SPD demonstrates superior generalizability, achieving 15% better performance under out-of-domain generalization settings.

自蒸馏大模型能力提升无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。