让模型学习能力与教师监督匹配,提升大模型推理能力。
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

- 构建统一优化框架,同时调节选词与信息强度
- 在多个基准上超越现有方法,小到70亿参数模型有效
- 轻量在线算法,自动匹配学习难度与监督强度
基于策略的自蒸馏(OPSD)通过将特权上下文内化到模型参数中,提升大语言模型的推理能力。现有研究分别关注选择学习哪些词或控制教师接收的特权信息量,但两者独立优化导致次优结果。本文指出,这两者通过学生的学习容量耦合:特权信息决定每词偏差,词权重则决定吸收哪些偏差。为此提出统一优化框架,最大化师生总偏差,受限于学生可承受的总学习难度预算。在此框架下,提出轻量级在线算法USD,用单一对偶变量同时确定词选择阈值和特权信息调整方向,使监督始终匹配学生演化中的学习能力。大量实验表明,USD在多种模型规模和推理基准上均优于OPSD及基线方法。代码已开源。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation. Two recent research lines promote vanilla OPSD by choosing which tokens to learn from and by controlling how much privileged information the teacher receives, respectively. However, we show that each line optimizes one variable while holding the other fixed, which leads to a suboptimal solution. We argue that the two variables are coupled through the student's learning capacity: the privileged information sets the per-token divergence the teacher prescribes, while token weighting selects which of these the student must absorb. We formalize the two lines of work into a unified optimization framework, which maximizes the aggregate teacher--student divergence, subject to a budget on the aggregate learning difficulty the student can absorb. Under this modelling, we propose Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian. USD reveals that a single dual variable governs both decisions: at one price for learning difficulty, it simultaneously sets the token-selection threshold and the direction of privileged-information adjustment, keeping supervision matched to the student's evolving capacity. Through extensive experiments, USD consistently demonstrates superior performance over OPSD and token- and PI-side baselines across various model scales on various reasoning benchmarks. Code is available at https://github.com/lauvlalala/USD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。