让大模型根据推理不确定性调整学习方向,提升数学推理能力。
Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning

- 根据注意力熵动态调整教师引导方向,高不确定时鼓励探索,低时强化准确。
- 在6个数学推理数据集上平均准确率提升至最佳,宏平均@16达最优。
- 适合需要复杂推理和稳定输出的场景,如AI数学助手、代码生成。
在策略自蒸馏(OPSD)中,模型自身充当教师,利用提示或参考轨迹对自身生成进行细粒度监督。但近期研究发现,这种统一方向的监督会抑制复杂推理中的预测不确定性,阻碍探索与假设修正。我们通过逐标记分析发现,问题源于对不同不确定性水平的标记使用相同的教师引导方向:高熵标记被强制服从教师,削弱探索;低熵标记偏离教师则降低步骤准确性。为此,我们提出方向自适应自蒸馏(DASD),将统一模仿重构为基于熵的定向监督:高熵标记被推离教师以保留探索性,低熵标记被拉向教师以稳定执行。在六个数学推理基准测试中,DASD在强基线(包括RLVR与自蒸馏)之上实现最佳宏平均@16。通过通过率@$k$、推理健康度与泛化性分析表明,这些平均提升来自在不牺牲步骤精度的前提下有效维持探索能力。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) is an emerging LLM post-training paradigm in which the model serves as its own teacher: conditioned on privileged information such as a reference trace or hint, the same policy provides dense token-level supervision on its own rollouts. However, recent studies show that OPSD degrades complex reasoning by suppressing predictive uncertainty, which supports exploration and hypothesis revision. Our token-level analysis shows that this failure arises from applying a uniform direction of teacher supervision across tokens with different uncertainty levels: conformity to the privileged self-teacher suppresses exploration at high entropy, while deviation from the teacher degrades step accuracy at low entropy. Accordingly, we propose \textbf{Direction-Adaptive Self-Distillation} (\textbf{DASD}), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision: high-entropy tokens are pushed away from the privileged teacher to preserve exploration, while low-entropy tokens are pulled toward the teacher to stabilize step-level execution. Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines. Pass@$k$, reasoning-health, and generalization analyses show that these average gains come from preserving exploration without sacrificing step-level execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。