让大模型在推理时更懂自己，提升思考效率与准确性
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
- 根据教师模型预测的不确定性动态调整训练信号权重
- 在Qwen3-4B/8B上实现更高推理准确率与更短思考路径
- 适合追求高效精准推理的大模型开发者
在策略自蒸馏中，模型基于自身生成的推理轨迹进行训练，同时由教师模型（常为带额外上下文的同一模型）提供细粒度的逐标记监督。现有方法对思维链中各位置的教师信号采用均匀加权，但忽略了教师预测分布熵的显著差异。本文提出EGRSD（熵引导强化自蒸馏），通过三个信号统一更新：奖励驱动方向、师生似然比大小，以及提出的教师熵置信门——对高熵标记位置降权，同时保持每位置权重非零下界。进一步提出CL-EGRSD，一种因果前瞻变体，可区分持续高熵段与短暂高熵位置（其后续上下文迅速转为低熵）。在Qwen3-4B和Qwen3-8B的思考模式下实验表明，EGRSD与CL-EGRSD在可训练方法中显著提升了准确率-长度前沿。
原文摘要 · Abstract (English)
On-policy self-distillation trains a reasoning model on its own rollouts while a teacher, often the same model conditioned on privileged context, provides dense token-level supervision. Existing objectives typically weight the teacher's token-level signal uniformly across a chain-of-thought sequence, despite substantial variation in the entropy of the teacher's predictive distribution. We propose EGRSD (Entropy-Guided Reinforced Self-Distillation), which unifies token-level updates through three signals: a reward-grounded direction, a teacher-student likelihood-ratio magnitude, and the proposed teacher-entropy confidence gate that down-weights high-entropy token positions while maintaining a nonzero lower bound on every token weight. We further introduce CL-EGRSD, a causal-lookahead variant that distinguishes sustained high-entropy spans from transient high-entropy positions whose following context rapidly becomes low entropy. Experiments with Qwen3-4B and Qwen3-8B in thinking mode show that EGRSD and CL-EGRSD advance the accuracy-length frontier among the compared trainable methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。