arXiv:2607.10805cs.CLcs.LG2026-07被引 3

解决大模型自蒸馏中思维崩溃问题,提升复杂推理准确率。

Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation

  • 通过动态调节蒸馏目标,防止学生模型在高熵决策点丢失推理能力。
  • 在数学推理任务上相比标准自蒸馏提升最高4.1%准确率。
  • 适合需要增强推理能力的大模型训练与对齐场景。

基于策略自蒸馏(OPSD)是提升和对齐大语言模型的重要方法,但在复杂推理任务中反而降低下游性能。本文系统研究该问题,发现一种称为‘思维崩溃’的优化陷阱——模型内生中间推理行为显著下降,表现为每千词的认知标记密度(ET per 1k)锐减。通过基于熵的梯度掩码与标记级目标分析,发现此现象由高学生熵决策分支处激进的教师梯度引发,导致学生认知标记频繁被压制为非认知目标,并高度集中于学生-教师差异大的区域。为此,提出自适应双视角自蒸馏(AD-OPSD),通过不对称点对点差异门控,将高抑制风险的标记锚定至冻结基础模型的参考先验,从而在保留错误纠正能力的同时保护原生思维能力。跨多个数学基准的实验证明,AD-OPSD在不同模型规模与数据集上相较标准OPS D提升最高达+4.1%绝对准确率。进一步分析显示,该方法有效缓解思维崩溃,并在多种后训练范式中具备强泛化性。

原文摘要 · Abstract (English)

On-Policy Self-Distillation (OPSD) has emerged as a crucial paradigm for enhancing and aligning Large Language Models (LLMs). However, in complex reasoning tasks, OPSD paradoxically degrades downstream performance. In this paper, we systematically investigate this pathology and identify a severe optimization trap we define as \textbf{Thinking Collapse} -- a sharp decline in the model's native intermediate reasoning behavior, measured by epistemic-token density (ET per 1k). Through entropy-based gradient masking and token-level target analysis, we show that this collapse is triggered by aggressive teacher gradients at high-student-entropy decision forks, where student epistemic tokens are frequently suppressed into teacher non-epistemic targets and are highly concentrated in high pointwise student-teacher divergence regions. To resolve this optimization pathology, we propose \textbf{Adaptive Dual-Perspective OPSD (AD-OPSD)}, a robust control framework that dynamically moderates the self-distillation objective. AD-OPSD selectively anchors high-suppression-risk sandboxed tokens to a reference prior derived from the frozen base model via an asymmetrical pointwise divergence gate, preserving native thinking capacity while retaining OPSD's error-correcting power. Extensive experiments across competitive mathematical benchmarks show that AD-OPSD improves over standard OPSD by up to \textbf{+4.1\%} absolute average accuracy across diverse model scales and datasets. Further analysis demonstrates that AD-OPSD mitigates thinking collapse and generalizes robustly to different post-training paradigms.

自蒸馏大模型推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。